Open Jev (intikhab49)

How we reviewed this: we ran it in python:3.12 in Docker on four CPU cores with no GPU — not the free T4 the README targets. We ran the full data validation, trained a small model end to end, calibrated it, evaluated it, served it and posted a schema it had never seen. There is no published checkpoint, so nothing here says anything about the quality of the author’s trained model; what we could check is whether the pipeline does what it says.

What it is

A 150M ModernBERT encoder with a marker token (<<l>>) inserted before every label in the prompt. One forward pass, one hidden state per marker, and a grouped softmax that normalises within each question rather than across the sequence. That gives you the Jev-shaped property that matters: the questions and their labels travel in the request, not in the weights, so you can add a question or rename a label without retraining.

We checked that claim rather than taking it. With a checkpoint we trained ourselves, we posted a schema it had never seen — an invoice email, a three-option choice and a noul — to POST /decide, and got a well-formed distribution per question back from one forward pass, 265 ms on four CPU cores. (The answers were near-uniform, which is exactly what a model trained for one epoch on 64 rows should produce; the mechanism is what was under test.)

Seven numbered scripts take you from baselines through training, calibration, evaluation, an ensemble and ONNX export, with a Colab notebook that embeds the whole repository.

The checks around the data are the best part

smoke_test.py --all validates every row of the dataset before a GPU is involved. On four CPU cores it took 30 seconds and reported:

ok  108 label positions all land on <<l>>
ok  grouped softmax normalises within questions only
ok  pad_cat joins ragged batches (widths [16, 17, 18, 20])
ok  encoded every row; 8000 questions checked, 0 label/gold mismatches

The README explains why each of those exists, and the reasons are the kind you only learn by being bitten: 10% of the dataset is a bare noul with no criteria key at all, so a six-row smoke test passes and training dies 300 rows in; the four workflows disagree on label width, so torch.cat fails the moment a run spans more than one; an unbounded temperature fit returned T=400, which minimises NLL beautifully and destroys the confidence signal.

That section — “Traps worth knowing if you fork this” — is more useful than most of the code in this directory.

What the numbers mean, and which one to read

When we first reviewed it, the header compared the ensemble’s 0.697 with Jev’s published accuracy, calibration, speed and cost. The Limitations section, further down, said something more precise, and it is the author saying it:

configuration	the author’s accuracy	schema
fine-tuned model alone	0.6240	dynamic
frozen probe alone	0.6705	fixed
ensemble (the headline)	0.6965	fixed

The ensemble blends in a frozen-encoder probe that only answers questions it has training labels for. On an unseen question the probe returns uniform and the blend degrades — so the number in the badge comes from the configuration that gives up the property the project is built around. The README states this plainly (“0.6965 is the known-schema number and ~0.624 is the new-schema number”), notes that Jev’s figure is zero-shot while this model trained in-domain, calls its own 400-case test split small, and says its Jev figures are quoted rather than reproduced. We did not reproduce any of them, and this review makes no comparison of its own.

Being that careful about your own headline is rare. We suggested moving the 0.624 up next to the badge, and on 6 October the author did: the README’s GitHub Action section now gives 0.6970 for the built-in workflows and 0.6235 for your own questions side by side, and says both are in-distribution.

One thing that used to be true and no longer is: 04_eval.py printed its Jev comparison from two constants at the top of the file, JEV_ACC and JEV_ECE_REPORTED, whatever you evaluated. Our 64-case dry run duly reported being “behind by 0.3864” against a full-split figure. Reported, and fixed in 5977431. We re-ran all three paths:

=== vs Jev 1.13.0: subset run, comparison suppressed ===
  This run covered 100 cases (--config customer_service).
  Jev's [published figures] are full-split figures over all 400
  test cases and all four configs - not a like-for-like baseline here.
  Run `python 04_eval.py` with no flags for the comparable number.

--limit 64 gets the same treatment, and only a run with no flags at all prints the delta. results.json now carries limit and full_split, so a subset run cannot be mistaken for a full one after the fact either — which was the quieter half of the report, and the half the author picked up on his own.

The released checkpoint

On 6 October the author published model-v1: an fp16 checkpoint (298 MB, in six parts with a SHA-256 to check after joining) and the ONNX files the project’s GitHub Action uses. The parts joined and verified. The release says to cast the weights back to fp32, and that the fp16 round trip changed no answer on 200 test decisions.

We ran it on the full 400-case, 2,000-decision test split, on the same four CPU cores, with the repository’s own scripts:

configuration	ours	the author’s (measured.json)
fine-tuned model alone, 07_ensemble.py	0.6235	0.6235
fine-tuned model alone, 04_eval.py	0.6190	—
frozen probe alone	0.6705	—
ensemble	0.6965	0.6970

The published numbers hold. The two model-alone rows use the same weights and differ by nine decisions between the two scripts; we did not trace why. Calibration is where our run and the author’s diverge. After its fitted sharpening the ensemble’s ECE was 0.091 for us, against 0.057 in measured.json, which the author measured through the Action’s onnxruntime path rather than these scripts.

04_eval.py’s coverage table is the useful one for routing: at a 0.70 confidence threshold the model alone answered 10.5% of decisions at 0.91 accuracy, and at 0.50 it answered half at 0.75.

Two practical notes. The checkpoint is a PyTorch pickle; on the current PyTorch, torch.load defaults to weights_only=True and it loaded under that, but check the SHA-256 first all the same. And the GitHub Action means you can now use it without training anything.

Training it yourself

Until 6 October there was no checkpoint, so the first thing anyone did was train. The README’s route is a free Colab T4, about 25 minutes. We took the slower road to see whether the pipeline holds up without a GPU at all.

On four CPU cores, 02_train.py --epochs 1 --limit 64 finished in under two minutes, 03_calibrate.py in 16 seconds, and 04_eval.py --limit 64 in a minute. The calibration step is worth watching: with soft targets it usually finds nothing to fix and says so — “temperature scaling did not help (0.1212 → 0.1212) — keeping T=1.0” — instead of silently applying a temperature that does nothing.

Scaled up, CPU stops being sensible. Three epochs over the full 1,200-row training split took us about two hours on four cores, with validation accuracy going 0.498, 0.527, 0.505 — best at epoch two, against the 0.483 majority-class baseline the repository quotes. On the official 400-case test split that model scored 0.5125 with an ECE of 0.0674, at 480 ms per case, and its coverage table was thin: 35% of decisions above 0.50 confidence at 0.66 accuracy, 5% above 0.70 at 0.89.

We also walked straight into one of the project’s own documented traps, which is the most useful thing we can report about it. To save CPU time we passed --max-len 512; the README says in bold not to, because states run 401 to 997 tokens and 512 truncates 73% of them. Three epochs and a truncated context produced 0.5125 where the author’s twenty epochs at 1,024 produce 0.624. The trap section was right, and it told us in advance exactly what we would lose.

The one number we did reproduce is the baseline: 01_ceiling.py gave us 0.670 for the frozen encoder plus logistic regression, against the 0.6705 in the README. That is the figure the whole ensemble rests on, and it came out on a CPU in a few minutes.

Permissions and maintenance

Nothing leaves your machine: the dataset comes from Hugging Face, the encoder from Hugging Face, and no key is required anywhere. 05_serve.py --serve bound 0.0.0.0 with no authentication; since 5977431 it binds 127.0.0.1 and takes a --host, and a non-loopback bind prints a warning first. We checked both, reading the listening socket rather than the log:

default (no --host)   LISTEN 127.0.0.1:8811   no warning
--host 0.0.0.0        LISTEN 0.0.0.0:8812     WARNING: binding 0.0.0.0 - /decide is
                                              unauthenticated and anything that can
                                              reach this port can query the model.

Apache-2.0, one author, and as of 5977431 there is CI: .github/workflows/smoke.yml runs compileall plus smoke_test.py --all on every push and pull request. That is the right job — it needs no GPU and no checkpoint, and it is what catches the two dataset-shape traps the README documents. We ran it offline against a warm cache: 8,000 questions encoded and checked, every label position landing on its marker, in about thirty seconds.

Verdict

Read this one for the write-up as much as the weights — and now there are weights. The data checks, the bounded calibration, the ONNX traps and the honesty about which configuration produced which number add up to the most instructive README in this group. The dynamic-schema property is real: we posted an unseen schema and got answers. And the released checkpoint reproduces the author’s accuracy to within a decision or two on a laptop-class CPU.

Both things we asked for are done: a checkpoint you can download, and the 0.624 beside the 0.697. The choice is still yours to make: the ensemble’s number needs the four built-in workflows, and your own questions get the model alone. For other open decision models you can run today, see OpenThai-SystemOne or Laya for Node; the rest are in the open models category.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Oct 7, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.