lorenzini

How we reviewed this: we ran its three test files in Docker with no network, then wrote our own stub gh and fed the shipping poller two review shapes we crafted — one with findings hidden in a collapsed section, one where the reviewer’s own count disagreed with what was posted — to see what verdict came out. We also ran it with the Jev shadow check enabled and no key at all. No Jev calls.

What it does

Three Claude Code skills poll a third-party pull-request reviewer — CodeRabbit, GitHub Copilot or Codex — and decide whether what it said amounts to a pass. The README is clear that this is not a code reviewer: “They do not review source code; they adjudicate what the reviewer reported.”

The rule the whole thing is built on is one line:

Never infer a pass from absence.

Every automated reviewer the author has watched puts findings somewhere the summary doesn’t count them — a collapsed <details> block, a comment thread, a bucket for findings that fall outside the diff. An empty result and a failed read look identical, and so do “reviewed, nothing found” and “the reviewer never ran”.

The ledger

docs/fail-open-ledger.md is 500 lines documenting eleven incidents where an earlier version of this tool returned an unearned pass, each with the repository, pull request number, commit and date, and closing with a section on what the pattern across them is. The framing is exactly right:

A gate that fails closed wastes a poll. A gate that fails open merges a defect and tells you it was fine.

It gets better. Entry 1 carries a note that the entry itself was false for two days: the cross-check it described as implemented was not in the code, and when it was added, the first implementation was dead — the literal is **Comments generated:** 3, with emphasis markers between the colon and the number, so a pattern for Comments generated: 3 matched nothing. A guard that cannot fire, written while fixing guards that did not fire, recorded as such in the document that lists them.

The same care is in the source. Almost every branch in the 1,100-line poller carries the pull request, head SHA and date where its absence was observed, including a measured transcript of three CodeRabbit review objects arriving three minutes apart, two of them empty replies that an earlier version had read as completion.

What we made it do

We wrote a stub gh and handed the real poller two reviews.

Findings hidden in a collapsed section. The review body says Actionable comments posted: 0 and then carries a <details> block titled “🧹 Nitpick comments (3)”. Zero inline comments. A summary-reading gate calls this clean:

RESULT=NITPICKS count=3

It printed the contents of the hidden section too.

The reviewer’s own count disagreeing with what was posted. Body says Actionable comments posted: 3; nothing was posted inline:

CodeRabbit reports 3 actionable comment(s) for this commit, but only
0 were found on it (0 open, 0 dispositioned).
Something it posted is not being counted. Do not read this as clean:
the gap is the finding.
RESULT=MISCOUNT claimed=3 counted=0

And two failures we caused by accident, which are the best evidence of all. Our first stub didn’t implement the GraphQL review-threads query. The poller said “Could not read the review threads. Retrying rather than counting zero findings” and ended at RESULT=TIMEOUT — not CLEAN. Then we ran the suite in a container without jq, and every script stopped at RESULT=ERROR jq not found. The one-line rule holds in the two places it is easiest to forget: a broken dependency and an endpoint that will not answer.

The verdict vocabulary is ten words wide — CLEAN, NITPICKS, SUGGESTIONS, PREMERGE, MISCOUNT, UNREPLIED, HOLD, OTHERBOT, TIMEOUT, ERROR — which is the right shape for something a merge decision hangs on.

Where Jev actually sits

This is the part worth understanding before you install it: Jev is not the gate. The classification is deterministic pattern matching, and the model runs opt-in (JEV_SHADOW=1) over every collapsed <details> label, to catch bucket names the hardcoded patterns don’t know, writing to a shadow log rather than into the verdict.

We confirmed that separation costs nothing when the model is unavailable. With JEV_SHADOW=1, no key and no network, the same review still produced RESULT=NITPICKS count=3, and jev.sh on its own exits 2 with “no key” and prints nothing else.

The question set lives in its own versioned file with a note we would quote at anyone building a classifier out of prompt text: “Changing one word here changes what this file classifies.” v1 scored 27/30 on the gold set and missed “Nitpick comments” at 0.37 because the word “nitpick” wasn’t in the criteria; one added sentence took it to 0.70. The file records its own measurements — 29/30, hidden-bucket recall 8/8, zero label flips over three repeats, dated — and the runner writes a new result file per variant rather than overwriting the comparison. Those are the author’s numbers; we did not re-run them, since that needs live calls.

jev.sh itself is 84 lines of curl and jq that pin the model (jev-latest “silently changes models, and a classifier’s version is part of its contract”), log a hash of the question set with every call, and read the key from the environment or a 0600 file — never printed, never passed on a command line.

Maintenance

Eighty-one commits in two days, 98 assertions passing offline, and tests that source the helpers from the shipping scripts rather than copying them, with a comment explaining why: “a copy drifts, and a test passing against a copy of the gate says nothing about the gate.”

There is no licence file, and gh reports none. For a repository whose whole value is a method other people should copy, that is the one thing to fix.

Verdict

The most carefully reasoned small project we have reviewed, and the rare case where the documentation is the artifact: read docs/fail-open-ledger.md even if you never install the skills, because the failure shape it catalogues — a filter that matches nothing being indistinguishable from a clean result — is in most of the tools in this directory.

Install it if you merge on bot reviews. Understand that Jev is the shadow here, not the judge, and ask the author for a licence before you build on it.

For the opposite arrangement, where the model is the gate, see Jevvy.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 21, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.