Jev Capability Atlas

How we reviewed this: we first ran all six suites in python:3.12-slim with --network none against an offline SDK stand-in at b85bae4, recomputing the published figures from the committed receipts. After the project rebuilt its history suite we re-checked at 4746cfd: we read every prompt the new suite sends, recomputed the new run from its receipts, and re-tested the defects that were still open. We made no Jev calls.

What it does

Not a leaderboard. Six suites, each probing one axis — is the answer fully present in the state you supplied? — with the raw per-case receipt committed alongside: probabilities, confidence, usage. All from synthetic or public data; nothing from your machine goes anywhere.

The discipline around it is unusually good. Each suite has a protocol.yaml that pins its parameters and annotates its own weaknesses — the triage suite declares annotators: 0 # labels are an existing pipeline's own historical dispositions, not independent annotation, and notes that its state is a fresh re-fetch rather than the original capture. The file header tells you to treat it as authoritative so README prose cannot drift away from it.

And it publishes a result that cuts against its own subject. The ICU alarm suite’s README says the run loses to the do-nothing baseline of keeping every alarm, and explains why: missed alarms are penalised five times over. That write-up also discloses the sampling method, the dataset licence, that the scale is deliberately naive, and a real bug the author found and fixed in feature extraction. We recomputed all four of its published metrics from the raw receipts and every one matched to the digit — notable, because none of them is produced by any script in the repository. They were worked out by hand, correctly.

The typo that was the finding

When we first reviewed this, the flagship history suite sent this to the model:

"state": "大清帝國的第二任皮帝是誰？ (1)康燕 (2)雍正 (3)乾隆"

皮帝 should be 皇帝 (emperor) and 康燕 should be 康熙 (Kangxi) — the correct option’s own name, misspelled in the text actually transmitted. The README built its headline on that case: Jev answering “wrong at 0.90 confidence” on a common-knowledge question. We reported the typos and a separate README error that paired two different questions as though they were one.

What happened next is the reason to read this project. Re-checking, the maintainer found that the items also had a counting-convention problem — “the 7th Qing emperor” has three different answers depending on whether you count from Nurhaci, Hong Taiji or Shunzhi, and the three options were one of each — withdrew the 2026-09-19 results entirely, and rebuilt the suite.

We verified the rebuild rather than taking it. The current data/cases.json has three scored cases with no ambiguity and no typos in any state, plus two unscored diagnostics that re-ask the old items with only the spelling fixed. Recomputing runs/2026-09-21.json ourselves, three repeats each:

case	asks	picks	confidence
A_bare_common	a common fact, no passage	3/3 correct	1.00
B_bare_obscure	an obscure fact, no passage	3/3 correct	0.87
C_with_context	the same fact, passage supplied	3/3 correct	1.00
D_v1A_typo_fixed	the old case, typo fixed	3/3 the same option	0.46–0.51
E_v1B_typo_fixed	the ambiguous one, typo fixed	2 / 2 / 3	0.07–0.13

Row D is the retraction in one line: with the spelling corrected, the confidently wrong answer is gone. Row E is the counting-ambiguity item behaving as an ambiguous question should — near-flat, and not even agreeing with itself across three runs.

The lesson the project now draws from the suite is a different and better one, stated in its own README: when the state itself is wrong, nothing flags that the question is odd — it still picks, confidently. That is a more useful warning than the one it replaced, and it came out of its own broken prompt.

The rebuild also added the things that make the next mistake catchable: three repeats per case, the sent state stored in every receipt, a --dry-run mode that prints the prompts so you can proofread before spending anything, and the withdrawn run kept in the repository for audit. Both README paragraphs that cited the old result now say what was withdrawn and why, and credit the report.

The licence is fixed too

At our first review GitHub reported NOASSERTION: the MIT text was followed by a rider saying it did not cover translated third-party material, and that “contributors are responsible for confirming that license permits redistribution before submitting”. That is now LICENSE as plain MIT — GitHub detects “MIT License” — with the scope notes moved into a separate NOTICE, and each translations/*/SOURCE.md carrying a reproduction-scope section that the template now requires. The maintainer’s own account of it was blunter than our report: most of those files had never filled in the licence field their own template asked for.

What is still open

The ICU suite’s regeneration pipeline and its runner disagree on a key. data/extract_features.py:162 writes "true_alarm"; run.py:39 reads c["expected_true_alarm"]. The committed cases.json uses the runner’s name, so it has been hand-edited since generation — regenerate the data and the run dies after it has already sent its first request. For an evidence base that is the defect that matters: the receipts are auditable, but the pipeline that made them is not re-runnable end to end.

Three smaller ones, all still present at 4746cfd: stage1-triage-filter/run.py:78 guards a zero denominator with or 1, so a threshold that never fires prints 0/1, which reads like “one attempt, none right” rather than “none at all”; the latency suite times from the first call with no warm-up discarded and takes an approximate rank for p95 at n=30; and scripts/common/jev_client.py:26 resolves .env.local from os.getcwd(), so running a suite from elsewhere can pick up another project’s key file. The shipped skill still contains https://github.com/REPLACE_WITH_REPO_URL.

The maintainer has listed all of these as open for contribution, noting none of them needs an API key — which is true; we reproduced each one offline.

There is still no CI and no test suite.

Verdict

The first version of this review said this was a project we wanted to exist, and that it needed a proofread, a corrected README and a pipeline that runs. Two of those three are done, and the way they were done is the point: a reader’s typo report was allowed to overturn the repository’s headline claim, the old results were withdrawn rather than quietly edited, and the receipts that made the check possible in the first place are still there.

Committing per-case receipts is the whole difference between an evidence base and a blog post. Read the suites rather than the summary, and treat a three-case suite as an illustration rather than a measurement — which is what its own protocol.yaml files keep telling you.

Its AGENTS.en.md says safety-critical irreversible gates belong in deterministic code — which is, exactly, the thing grok-bot-jev gets wrong.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 21, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.