jeval

Checked, not fully reviewed. We ran its 647 tests (all pass, two skipped) and its demo inside a container with no network, and ingested a Jev-shaped response log. jeval makes no model calls at all, so neither did we.

You give it labelled decision records — one per question, with the model’s prediction, probabilities and the right answer. You get back one offline HTML report and a thresholds.yaml your app reads: how often a 0.9 is actually right, where confidence stops separating right answers from wrong ones, and, given a cost matrix, the threshold that minimises expected cost per action, with a 95% interval and the share it automates. It keeps the model string from each response, so jev-latest quietly becoming a new version shows up as drift. With too few labels it says so and plans how many more it needs. The demo’s report flagged wide intervals and said “more labels, not more analysis” would fix them.

One thing to know before you wire its output into a Jev app. jeval’s confidence for a choice is the top probability, and its thresholds are computed on that. Jev’s response field called confidence is a different number: (p_max − 1/n)/(1 − 1/n), as TypeSafe publishes. The jev-native ingest preset gets this right on the way in. A choice at 0.6/0.3/0.1, which Jev reports with confidence: 0.4, was stored as 0.6. But thresholds.yaml says only threshold: 0.84, and the natural line of app code is answer.confidence >= 0.84. On a four-option question that means a top probability of at least 0.88, so the app automates less than the report predicts. Compare the threshold with the largest value in probabilities instead. We reported it.

Fixed on 13 October (#12, 31ebc7d). We re-ran the demo and jeval threshold at that commit. Every action in thresholds.yaml now carries a comparison block naming the value to compare for each question type: the chosen option’s probability for a choice, max(p, 1 − p) for a noul, and the reported confidence for a score. The CLI prints the same rule when it writes the file. The numbers are unchanged. The README now explains why Jev’s choice confidence is a different scale, and that it has no single equivalent threshold when option counts vary. The fix is on GitHub. The PyPI package is still 0.1.0 from 20 September, so install from the repository to get it.

Apache-2.0, 22 stars, Python 3.10+.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Oct 18, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.