Valen

How we reviewed this: we installed the repository at 3b764c9 with PyTorch’s CPU build, ran its 215 tests (all pass, 9 skipped), downloaded the published checkpoint and its pinned base model, and evaluated it with the project’s own valen.evaluate on samples of the project’s own evaluation sets, inside a container with no network. Numbers below are ours, on four CPU cores. The README asks for an NVIDIA GPU, so our timings say what a laptop gets, not what the project claims; its own latency figures and its comparisons with a 27B model are the author’s and we have not reproduced them. Valen makes no Jev calls.

What it is

A Qwen3.5-2B vision-language backbone with a LoRA, a few retrained vision blocks and one shared decision head. You send a state — text, images or video as chat messages — and typed questions; it returns a probability for every option instead of generating text. Choice and Noul take one pass per question; a Score takes one backbone pass per level, so a ten-level score costs ten passes.

The answer types are Jev’s, and so are the confidence formulas: Choice uses (p_max − 1/n)/(1 − 1/n) and Score the distance-from-mode formula TypeSafe published on 3 October. We recomputed the docs’ worked example, (0.1, 0.3, 0.6) → 0.25, and it matches. That makes its confidences comparable with Jev’s, which few open projects can say.

What it does not have is a server. Inference and evaluation read a JSONL file and write one, so code written for /v1/systemone cannot point at it; you would wrap it yourself.

The checkpoint you can download

The repository publishes one checkpoint, Valen-Preview-0923. Its model card is clear about what it is: a “2B Sokoban-trained preview”, reinforcement-tuned on 30,000 Sokoban boards from a general starting point, and “the General results are not measurements of this Sokoban preview”. The GitHub README is less clear. It shows one results figure, General and Sokoban side by side, and only the figure’s alt text says the General panel used “the General-trained RL checkpoint”, which is not on the Hub. Read the README alone and you would expect the download to be the model in the General panel.

On its own game it holds up. On every fifth record of the project’s 500 single-step Sokoban questions — 100 boards — it picked the right move on 80 of the 89 with a single correct answer (89.9%; the card reports 87.6% on all 500), and one of the optimal moves on 91 of all 100. Each board took a median 24 s on four CPU cores, so this is GPU software in practice.

Off its game it is a different model. The repository’s own smoke file, the one the README’s quick start runs, asks about traffic lights. For “The traffic light is red.”, the choice between red, green and unknown came back unknown at 0.82, with red at 0.06. All four traffic-light records — text, image and video — answered unknown.

We then sampled the project’s General evaluation set, 45 records across its ten sources. Fifteen did not finish within three minutes each on our CPU, all of them large images — photos of signs, desktop screenshots, documents — so the 30 that did are biased towards small pictures:

question type	right	note
multiple choice	19 of 22	mostly GQA and VQAv2 photos, four options
yes/no	3 of 8	answered “no” all eight times; the five whose answer was “yes” were all wrong

A model that says “no” to every yes/no question is not one you can gate on, and the smoke file shows the same lean: “crossing is allowed only when the light is green”, asked about a green light, came back at 0.49, 0.40 and 0.25 for text, image and video. This is a small sample, and it is the Sokoban checkpoint doing work it was not tuned for — which is the point. It is the only one you can download.

Two things before you run it

The quick start fails. Downloading the base model with hf download, as the README says, and then running inference raises “Base model revision differs from checkpoint”. The revision is in fact the same. The real problem is a missing valen_manifest.json, which scripts/setup/prepare_model.py writes and the README skips. Pinning the base model by commit and weight hash is good practice; the message just names the wrong cause.

The checkpoint is a 1.2 GB PyTorch pickle loaded with weights_only=False, and a pickle can run code when it is loaded. A comment beside the call says it is meant for “trusted local storage”, but inference uses it on the file downloaded from the Hub. We opened the published file without loading it: it references only tensor and storage types, and it loads with weights_only=True. So it is safe today, and the safer setting would cost nothing. We reported both.

The datasets are tagged Apache-2.0 on the Hub. Their cards then list sources under other terms, including ChartQA under GPL-3.0 and VQAv2 under its own terms, and say the list is “not an independent verification of upstream licensing”. If you reuse the data, follow the per-source terms.

Verdict

The most complete open attempt we have seen at a Jev-style model that looks at images. The training code, data, recipes and evaluation harness are all published, the confidences use TypeSafe’s formulas, and the base model is pinned by hash. 637 stars, Apache-2.0.

Take the release for what its model card says it is: a Sokoban specialist and a training kit. If you want a general visual yes/no model today, the checkpoint behind the General panel is not yet downloadable, and the one that is says “no” too often to use for that. For text-only decisions on a CPU, see jevos; for another open decision model, Laya.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Oct 6, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.