OpenJevPro

How we reviewed this: we read all 599 lines, drove the client against a stand-in Ollama endpoint that logged every prompt so we could see what reaches the model, and recomputed its published comparison table from the results file committed beside it. We made no Jev calls. The Jev figures discussed below are the author’s, from the author’s run; we do not republish other projects’ Jev comparisons, and nothing here is a measurement of Jev by us.

What it is

Not a model. OpenJevProClient builds a prompt that lists your candidates and asks an Ollama or OpenAI-compatible model to “rate the relative likelihood score (0.0 to 10.0) of each candidate” as JSON, then runs those numbers through a softmax. That is the whole mechanism, and at 599 lines across five files it is easy to read.

The repository describes itself as a “Production-Grade Open Alternative to TypeSafe Jev”. Two words there need qualifying.

“Open”: the LICENSE is PolyForm Noncommercial 1.0.0. Free for personal learning, academic research, non-profits and non-commercial development; any commercial enterprise, production environment or paid service needs a separate licence from the maintainers, as LICENSE-COMMERCIAL.md sets out. The README’s licence section says this plainly — it is the repository description and the “Open Source / Zero-Shot” cell in its own comparison table that do not.

“Production-Grade”: when we reviewed it there were no tests and the only GitHub workflow published to PyPI. There are six tests now, added with the fixes below; they cover exactly what was reported and run offline. No workflow runs them.

What “calibrated” means here

TemperatureCalibrator divides the model’s self-reported scores by a temperature and softmaxes them. The temperature is temperature_scaling: float = 1.25, a constructor default. Nothing fits it — grep finds one construction site in the codebase and no labelled data anywhere near it.

So “mathematically calibrated posterior probabilities” is a fixed divisor applied to numbers a language model wrote in a JSON field. Whatever that produces, it is not a posterior, and dividing by 1.25 cannot make it one.

The abstention column is built by the harness

openjevpro/harness.py runs three engines over the same items. Here is what each one sends.

Its own engine:

decision = self.client.decide_choice(..., allow_abstain=True)

and in the client:

if allow_abstain and "UNKNOWN" not in options:
    options.append("UNKNOWN")

The comparison engine:

payload = {"model": self.model, "state": state_repr,
           "questions": {"decision": {"type": "choice",
                                      "instructions": "Select the single best category matching the state context.",
                                      "criteria": criteria}}}
...
"abstained": ans.get("choice") == "UNKNOWN",

The criteria dict goes out unchanged, and the result is then tested for an option that was never in it. We confirmed this on the wire, driving the real client against a stand-in that logged the prompts:

request 1 (allow_abstain=True):  ['card_arrival', 'change_pin', 'top_up_failed', 'UNKNOWN']
request 2 (allow_abstain=False): ['card_arrival', 'change_pin', 'top_up_failed']

So the README’s row — “Out-of-Scope Rejection: 100.0% (Safe Abstain)” against “0.0% (Forced Pick)” — reports that a system offered an escape hatch used it, and that a system denied one did not. The six out-of-scope cases are also the entire accuracy gap in the table above it.

Giving one side an abstain option is a legitimate design choice and the paper-shaped version of this comparison would be interesting. Scoring the other side as 0% for not returning a label it was never offered is not.

Fixed on 2026-09-23. allow_abstain=True now injects UNKNOWN into the criteria for every engine in the harness, and the author re-ran the whole set against the live API rather than editing the table. All three engines now reject all six out-of-scope cases, which is the result the symmetric version of this comparison was always going to produce and is the reason the column stopped being interesting. There are also six tests now, where there were none — they pin the abstention contract, symmetric injection for both comparison engines, the ECE computation and the new selective metrics, and they run offline in 0.3 s.

The table did not reproduce from its own file. Now it does

examples/harness_benchmark_results.json holds all 36 records with per-engine choices, confidences and latencies, so the published table can be checked rather than believed. We recomputed it:

README’s “Fresh Empirical Replication”	what the committed file gives
in-domain accuracy 100.0% (29/29), both engines	there are 30 in-domain cases; OpenJevPro 30/30, the other engine 29/30
OpenJevPro P50 latency ~508 ms (Cloud 248ms)	4979 ms
the other engine’s P50 ~777 ms	772 ms
“faster end-to-end response times (~508ms vs ~777ms)”	its own file makes it 6.4× slower, not faster
ECE 0.089 vs 0.284 vs 0.312	no ECE field exists in either artifact; recomputed over 10 bins from the per-record confidences we get 0.004, 0.075, 0.035

Three separate problems. The in-domain cell borrows the overall correct count (29) into a denominator it does not belong to, and in doing so asserts a perfect in-domain agreement that the file contradicts — there is one in-domain miss in the records, on “How do I locate my card?”. The latency row looks like 5081 (the mean in the file) with a decimal point moved, and the conclusion drawn from it is the reverse of what the data says. And the calibration column has no backing artifact at all; our recomputation puts every engine better-calibrated than claimed and changes the ordering.

The summary block inside the JSON is internally consistent — accuracy: 80.56, p50_latency_ms: 4979 — so the harness computed the right things. The README is where they went wrong.

The re-run, checked the same way

The author replaced the artifacts and the table on 2026-09-23 after a live re-run, so we did the arithmetic again at 1045f42: every field of the summary, recomputed from the 36 records, for all three engines. Accuracy and correct counts, selective accuracy, tentative accuracy, coverage, abstained counts, the in-domain and out-of-scope splits with their denominators, mean and P50 latency, and 10-bin ECE — all of them match to the decimal place published.

One field did not match our arithmetic, and it was a convention rather than an error: P95 latency. Their figure is sorted(latencies)[int(0.95 * n)], which for 36 samples is the second-largest; nearest-rank gives the one below it.

That is documented now. 20e1fa4 prints the convention in the README and publishes all three readings side by side — ceil-rank, nearest-rank and linear interpolation — and the numbers match ours to the decimal place. The same commit added a CI workflow that runs the tests on three Python versions (they were in the repository but nothing ran them), relabelled the first comparison table’s row as an independent reference to openjev-sglang by @ekzhang rather than as this project’s own result, and changed the self-description to “Source-Available”, which is what PolyForm Noncommercial is.

The in-domain denominator is right this time (30 in-domain cases, counted as 30), the latency row matches the file rather than inverting it, and the ECE column now has an artifact behind it and a test for how it is computed. The numbers in that table are the author’s, from the author’s run on a live key; what we checked is that the table and the records agree, which is the thing that was wrong before and is the only thing an outside reader can check.

One more from the code

ChoiceDecision can come back with abstained=True while value still names a real category:

choice='card_arrival' confidence=0.250 abstained=True

abstained was set when the winner was UNKNOWN or when confidence fell under abstain_threshold (0.45), but value was left as the argmax either way. A caller that read .value and not .abstained got a plausible-looking label for a case the engine had declined to decide.

That is fixed as well: an abstained decision now returns value="UNKNOWN" and moves the argmax to a separate tentative_value, which is also what the new tentative_accuracy column reads. The threshold gained an "auto" setting that scales as 1.25 / n_options, for the multi-class case where a softmax over seven candidates cannot reach 0.45 however clear the answer is.

On the first table

The comparison above the replication cites third-party figures from an external leaderboard, and labels one row “OpenJevPro (openjev-sglang)”. openjev-sglang is Eric Zhang’s project — 259 stars, a different repository, which we have reviewed separately. Whatever that row measures, it is not this code.

Verdict

The idea underneath — add an explicit abstain option, then threshold on confidence — is sound and is what several better projects in this directory do. The headline table did not demonstrate it: the option went to one side only, and three of its columns disagreed with the file checked in beside them. All of that was fixed within two days of our report, by re-running rather than by editing, and we have checked the new table against the new records field by field.

What is left is the calibration. TemperatureCalibrator.fit() now exists — bounded NLL optimisation over labelled logits, and it works: on deliberately noisy labels it pushed the temperature from 1.25 to 9.33 and flattened a 4.0-versus-0.0 logit gap to 0.61/0.39, which is the right answer for labels that carry no signal. But the client still constructs the calibrator with temperature_scaling: float = 1.25 and nothing in the decision path calls fit; the only callers are its tests. So the number you get out of the box is still a fixed divisor, with a supported way to replace it if you have labelled data. The licence is still PolyForm Noncommercial — described accurately now.

If you want typed decisions from an open model you host, AnyJev reads the option logits and publishes the runs where its method loses, and openjev-sglang serves the API shape on SGLang. If you want this one, note the licence before you put it in anything commercial.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 24, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.