openJev-verdict-2.0

How we reviewed this: we ran its 42 tests in Docker on four CPU cores, recomputed its published metrics from its own prediction log, rebuilt its evaluation set from the public dataset, and ran its training pipeline end to end. We made no calls to TypeSafe.

What it is

Two stacks in one repository, and telling them apart is most of the work.

verdict2/ is what the README describes: ModernBERT-base with a marker-pointer head and a separate correctness head, 149,614,210 parameters by our count, which matches their figure. core/ and rlcd/ are an earlier system built on GLiClass — and that is what import openjev actually exposes.

You cannot get the model

The checkpoint the README is about, artifacts/verdict2-base/model.pt, is a 134-byte Git LFS pointer. We asked the LFS server for the object:

{"oid":"2201b07c…","size":598509338,
 "error":{"code":404,"message":"Object does not exist on the server"}}

The README’s badge points at huggingface.co/heman10x/openJev-verdict-2.0, which returns 401 — we checked; the same account’s rlcd-modernbert-151m returns 200, so this is not a network problem at our end. Following the runbook’s evaluation step gives:

_pickle.UnpicklingError: invalid load key, 'v'.

'v' is the first byte of version https://git-lfs.github.com/spec/v1.

What the download script does fetch is the v1 GLiClass model, and its bundle manifest says so: "model_architecture": "knowledgator/gliclass-modern-base-v2.0", 151,378,177 parameters, trained in 645 seconds on Apple Silicon. That is not the 149.6M model trained for 8.8 hours that the README describes.

Driving that v1 model on an unambiguous duplicate-charge incident, it abstained on the choice question at 0.95 and on the score question at 0.98; only the yes/no answered. Their own file reports/reference_floors.json records the v1 baseline at 0.261 accuracy against a 0.2985 uniform floor, and the runbook says, in its own words, that being reliably worse than chance “points at a wiring defect”. Worse, the bundled example script defaults to a directory the download script doesn’t write to, so it silently runs the un-finetuned base model and prints a confident answer to a question about a solar eclipse.

The parts that are genuinely good

It would be easy to stop there, and it would misrepresent the project.

• Every published metric recomputes. We ran their metrics.py over their own 2,000-row prediction log and got their accuracy, Brier, both ECEs, both AUROCs and all four coverage points to the digit.
• The evaluation set checks out independently. We rebuilt the items from the public LocalLLaMA/typed-decisions dataset: 2,000 of 2,000 predictions matched, zero soft-target mismatches, zero gold-label mismatches. Nothing was dropped or re-labelled.
• The training pipeline runs. A smoke run on four CPU cores completed in 338 seconds including temperature fitting and evaluation, and cleared the runbook’s own floor.
• Artifacts are SHA-256 verified on download, with a non-zero exit on mismatch. We have reviewed a lot of projects this month; this is the first to do it.
• Negative results are kept. The shipped calibrator records that the fitted temperature made things worse on NLL, Brier and ECE, and that 1.0 was shipped instead.

One caveat to carry if you compare numbers: their Brier is computed against a soft teacher distribution rather than one-hot truth, and score questions are excluded from it.

What we measured

Four cores of an i3-9100T, no GPU, 30 timed calls after warm-up:

• v1 GLiClass, one question: 81 ms median, 1.10 GB resident, 5.5 s to load.
• The Verdict 2.0 architecture, from a checkpoint we trained ourselves because theirs is unavailable: 79 ms median, 1.66 GB resident. Accuracy from that run means nothing; the shape and cost are real.

Batching is not free: three questions in one call took 185 ms against 81 ms for one. The per-question latency_ms in the response is the total divided by the number of questions, not a measurement.

The licence is not Apache-2.0

GitHub reports NOASSERTION and the badge says Apache 2.0. GitHub is right. The LICENSE file is 410 words against the real licence’s 1,581, and what is missing is section 4 in its entirety — the requirement to pass on the licence, to mark modified files, to retain attribution notices, and to carry the NOTICE file. The patent-termination clause and most of the disclaimer wording are gone too, and no copyright holder is named.

Everything upstream is properly Apache-2.0, so there is no conflict to resolve — but nobody can rely on this file, and a compliance review will stop at it.

Also worth knowing: torch.load(..., weights_only=False) appears in three places including the documented evaluation entry point, so pointing it at someone else’s checkpoint runs their code. Their own documentation-freshness test fails, reporting that the README’s generated-table markers have been removed — meaning the numbers in it are now maintained by hand, and the mechanism built to keep them honest has been bypassed.

Verdict

The engineering habits here are better than almost anything else we have reviewed: checksummed downloads, an auditable prediction log, a public evaluation set that reconciles row by row, a runnable pipeline, and negative results left in the repository.

None of that can be pointed at the model it is about, because that model is not published. Until the LFS object or the Hugging Face repository is restored, what you can actually run is a v1 system that its own report marks as worse than chance. Reported, along with the licence.

For local models you can download today, see SemIf, NanoJev and LocalJev.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 20, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.