jev-test-filter

How we reviewed this: we built a scratch repository with twelve tests, made a real commit, and drove the built CLI against a stand-in whose score for each test we chose — then broke that stand-in six different ways. We made no Jev calls.

What it is

It reads a git diff, asks one question per test about whether that change can alter that test’s outcome, and prints the filter arguments your runner already understands — vitest, jest, node --test, Playwright, cargo test, go test. One shared state, one round trip, no agent loop, and the model never reads a file.

The sentence that decides everything

From run.ts, above the function every failure path calls:

/**
 * A selection is an optimization and never a correctness gate, so every
 * failure below takes the same exit: run everything, and say why on the
 * result rather than in a log nobody reads.
 */

That is the correct reading of the risk. A test selector that fails closed skips the test that would have caught the bug, and you find out in production. So we spent most of our time trying to make it skip something.

what we did	what it did
provider returned HTTP 500	running everything (jev failed: HTTP 500: boom) — 6/6
provider returned HTML	running everything (jev failed: … is not valid JSON) — 6/6
connection refused	running everything (jev failed: network: …) — 6/6
no API key at all	running everything (jev failed: no API key; set TYPESAFE_API_KEY) — 6/6
--dry-run	running everything (--dry-run: no questions were asked), 0 requests
answers in the wrong shape	the tests with unusable answers were kept

Every one of those is on stderr and in the fallback field of --json, so a CI step can see it without parsing log text. That last row matters as much as the others: a per-test answer that cannot be read keeps that test, rather than dropping it into the gap between “no” and “unreadable”.

The guard we did not expect

There is a second failure mode that most tools would never think of, and it has its own check:

// A truncated diff that still deselects most of the suite is a selection
// made from a state that was missing the change it should have judged.

If the diff did not fit the state budget and the selection dropped more than half the suite, it throws the selection away. We reproduced it by committing a 120,000-line file:

selected: 12 / 12
fallback: "the diff did not fit the state budget and the selection was small"

The model answered. The answers were well-formed. The tool discarded them anyway, because it knew the model had not seen the whole change. That is a failure nobody would have filed a bug about — the run would simply have been quietly wrong — and somebody thought of it in advance.

Worth knowing the exact boundary, since we found it by accident: the condition is strictly less than half. A run that truncated and selected exactly 3 of 6 did not trigger it.

When it works

Given a commit that touched only parseFlags, and a stand-in that scored the three parseFlags tests at the top level and everything else at the bottom:

mode: pattern   selected: 3 / 6
argv: ["--test-name-pattern",
       "^(?:parseFlags reads a single flag|parseFlags ignores positionals|parseFlags handles an empty argv)$",
       "test/parse.test.js"]

The right three, a correctly anchored and escaped pattern, and the file. The four score levels it offers are worth reading, because the middle two are where the judgement actually lives:

0 — Unrelated to this change: running this test cannot produce a different result than before. 1 — This test exercises code the change touched, but nothing in the change can alter its outcome. 2 — This test could be affected by the change: it might fail. 3 — This test directly exercises behaviour the change altered or broke: it is likely to fail.

Level 1 is the distinction most selectors miss — touching the code is not the same as being able to change the answer — and the default cutoff of 2 means a merely-adjacent test is dropped while an at-risk one is kept.

Offline afterwards

--replay re-grades a recorded run at today’s cutoffs, with no key and no network. We ran it with TYPESAFE_API_KEY unset and it reproduced the same 3-of-6 selection from .jev-test-filter/last.json. So you can change your cutoff and see what it would have selected without paying again — and the gating code is deliberately kept outside the network try, so “a bug in the gate must not be reported to the user as a network failure”.

Engineering

MIT, Node 24+, git required. 107 tests pass and tsc --noEmit is clean. The three runtime dependencies are @ast-grep/napi and its Rust and Go grammars, used to extract test names from source rather than by running the suite first.

Fifteen stars, one author, version 0.1.0, and the repository was one day old when we tested it.

Verdict

The design decision at the top of run.ts is the whole review: this thing knows what it is for, and it knows that being wrong in one direction is free and being wrong in the other direction is expensive. Every failure we could manufacture landed on the free side.

Use it if your suite is slow enough that running all of it on every pull request hurts. Read run.ts either way — the truncated-diff guard is the kind of thinking most tools in this directory have not done yet.

The obvious caveat remains: a test that is mis-scored is a test you did not run, and no amount of failing open fixes a confident wrong answer. That is what jev-calibrate is for, and running one against the other would be a good afternoon.

For a gate that refuses a “done” instead of selecting tests, see Canny.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 22, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.