jev-benchmarks

How we reviewed this: we ran its tests in Docker offline and compared the published report against the code that produces it. We ran no benchmark and make no statement about any result — our policy is not to publish accuracy, latency or cost figures for the service.

What it does

jev-benchmarks, by AbdelStark, compares Jev with an open zero-shot classifier on the one thing both can do: assign a single label and give a probability per label. It measures discrimination, probability quality (Brier, NLL, calibration error), how much coverage you can automate at a fixed error budget, system behaviour, and bootstrap intervals — with a protocol document written before the results.

The engineering habits are good: a pinned dataset revision, deterministic class-balanced sampling, append-only prediction files so a run resumes, hashes of every artifact, and failures recorded as an explicit non-prediction that still counts against accuracy.

One design detail worth noting: the Jev adapter uses meaningless option keys and puts the label text in the descriptions, while the other backend receives the label strings directly. That’s a deliberate choice about what each system is asked to read.

39 tests pass offline.

Auditable, not reproducible

You cannot re-run this without network access and a key: the dataset downloads at preparation time, the model checkpoint downloads on first use, and the per-example predictions are gitignored. What ships is the hashes. So a third party can check that the published numbers correspond to files with those hashes — but cannot recompute them.

We did verify that the published report’s configuration hash matches the configuration file in the repository, and that the per-dataset metric keys match what the scoring code emits. The outer envelope has been hand-tidied relative to what the code writes, which the report itself partly explains.

Two gaps between the protocol and the code:

• The protocol pre-registers a primary outcome — macro-F1 averaged equally across datasets — and nothing in the source computes a cross-dataset average. The published results are per-dataset secondary metrics.
• The protocol says a uniform-probability predictor is included as a negative control. The code computes it; the published results contain only the two real backends.

Also worth flagging for contributors: the README’s minimal install produces a test suite that won’t even collect, because a test imports an adapter whose dependency lives in the other extra. CI uses the fuller install.

Verdict

As research engineering, this is the most disciplined evaluation project we’ve seen in this ecosystem — the protocol, the hashes, the resumable runs, the refusal to let failures vanish.

Read it as an audit trail, not a replication package, and note that the metric it promised to lead with isn’t in there.

For calibrating on your own data instead, see jevcal.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 20, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.