jevcal

What it does

jevcal answers the question every Jev integration eventually runs into: at what confidence can I let Jev act on its own? You describe your questions in a YAML file (the same shape as the TypeSafe API), bring a few hundred labeled examples, and jevcal:

1. runs Jev over your data,
2. picks a threshold per question that meets your accuracy target,
3. verifies each threshold on a held-out half of the data before trusting it,
4. writes a lock file and an HTML report, and
5. gives you a small runtime that uses Jev when it’s confident and falls back to an LLM, or to “no answer”, when it isn’t.

It’s the automated version of our confidence thresholds guide, and it follows the same rule we do: measure privately. The README says plainly that jevcal “ships no leaderboard and publishes nothing.”

Setup

It isn’t on PyPI yet, so you install from GitHub:

pip install "git+https://github.com/abhixhek/jevcal"
jevcal demo

It needs Python 3.10 or newer and has a single dependency, PyYAML. It talks to the TypeSafe API over plain HTTPS rather than the SDK, and TYPESAFE_BASE_URL lets you point it at a gateway. The demo runs the whole workflow against a built-in simulator, no key required, and the simulator is deliberately overconfident so the report has something to show. All 24 tests passed in our run, and the project’s CI tests three Python versions and runs the demo on every push.

Using it

We ran the full workflow on the bundled example (400 synthetic support tickets) with a real Jev key. As with all our reviews, we report how the tool behaves, not how Jev scored.

• lint is a useful first pass. On a set of deliberately bad questions, it flagged a double negative as an error and warned about a counting question, a compound “urgent and about billing” question, and too-short instructions, each with a concrete fix. It missed a date comparison phrased as “before the 2026-09-01 deadline” and two options whose descriptions overlapped in meaning but not in words. The rules are pattern-based, so treat a clean lint as a good sign, not a guarantee.
• run produces exactly what it promises. A lock file with the threshold, target, and status for each question, plus hashes of the dataset and questions, the model version that actually answered, and the random seed, so results are reproducible. The HTML report has a section per question.
• It’s built not to flatter you. The accuracy and coverage figures it prints are measured on the held-out half, not the half used to pick the threshold, and the output says so. Per the README, a threshold that doesn’t hold on that half is reported as such rather than hidden. All three example thresholds held in our run, so we didn’t see that warning ourselves. --conservative goes further and requires the lower confidence bound to meet your target.
• check catches drift. Against the same data it passed. After we corrupted about 30% of one question’s labels to simulate a shift, it exited with an error and the message “recompile thresholds before trusting this model version”. --strict also fails when a different model version answers than the one you tuned on.
• The runtime fails closed. With no fallback configured and thresholds that can’t be met, every answer came back as unresolved with no value, never as a guess.

We didn’t test label (an LLM fills in missing labels), optimize (an LLM rewrites weak questions and the held-out data decides), or the LLM fallback, since they need an Anthropic, OpenAI, or OpenRouter key. The author says those paths are covered by tests with a stub LLM. There’s also a claude-cli: option that uses your local Claude Code login instead of an API key.

Permissions and data

• Your dataset goes to TypeSafe, one request per row, which is inherent to measuring Jev.
• LLM features send data to a second provider. label, optimize, and the runtime fallback send rows to whichever LLM you configure. Check that it’s approved for your data.
• Responses are cached locally in .jevcal/cache in your working directory, so reruns are free. In our run the cache held Jev’s responses, not the raw ticket text. The project’s own .gitignore excludes it; add .jevcal/ and jevcal-report.html to yours too, since the report and lock file describe your model’s performance.
• It reads TYPESAFE_API_KEY from the environment and doesn’t write it anywhere.

Maintenance

jevcal was created on September 18 and had four commits and a handful of stars when we tested it. The code quality and test coverage are well above that level, and CI is set up, but it’s a single-author project that hasn’t published a release. Pin the commit you use.

Verdict

If you plan to let Jev take real actions on its own, run jevcal first. It turns “the right threshold depends on your data” into a repeatable command, is designed not to overstate results, and gives you a CI guard for the day jev-latest moves. Its main weaknesses are the ones you’d expect from a days-old project: no PyPI release, one maintainer, and a linter that catches common mistakes but not every rewording.

For other ways to evaluate Jev on your own data, see Janus and jev-benchmarks in the evaluation category.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 19, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.