jeff (Firelex)

How we reviewed this: we ran the v1.2 base and two of its nine adapters on CPU in Docker, replayed the exact 300-row test samples the project’s own benchmark used, and recomputed its headline table from the published result files. We could not run the 27B it compares against — it needs 28.6 GB — so that side of the comparison is the author’s. No Jev calls.

What it is

Jeff is an open 0.8B “System 1” model on Qwen3.5, served by jeff-serve at /v1/systemone. Questions are noul, choice (up to 254 options) and score, answered from one forward pass with a probability per option. On top of the zero-shot base sit LoRA adapters, one per job: guard (prompt injection), triage, support intents, tools, grounding, voice navigation, spam, legal clauses, emotion. A request names an adapter as its model; anything else goes to the untouched base. Each adapter ships its test and calibration sets, and a decision_config.json that records the SHA-256 of its training data and of every training script. MIT, 1,351 stars in six days.

The headline, recomputed

The README leads with Put Jeff in front of your 27B: let Jeff decide, and pass a decision to Qwen3.8-27B only when Jeff is unsure. Across eight adapters the author reports 95.3% accuracy for the cascade against 86.6% for the 27B alone, in 0.25 s instead of 8.1 s.

Those numbers come from jeff-reference-app, and they hold. We recomputed every mean from results/cascade.json — 86.6%, 95.3%, 8.11 s, 0.25 s, a 37.7× geometric speed-up — and checked the two things that most often flatter a comparison like this. The 27B’s 300 replies per task contain no invalid JSON, so Jeff is not winning on format failures; and the test and calibration rows come from disjoint families, with no state shared between them, and thresholds chosen on calibration rows only. The 27B run records its exact revision, 8-bit quantisation, machine, thinking: off, sample seed and a hash of its row ids.

Then we replayed those same 300 rows through Jeff on our CPU:

adapter	author reports	we measured
guard	98.0%	98.0% (294/300)
triage	91.0%	91.7% (275/300)

Guard reproduces exactly. Triage is two rows better than published.

What the table does not say

The cascade’s threshold, per task, is the lowest one whose accuracy beats the 27B alone by a point. On seven of the eight headline tasks that threshold is 0.00, so nothing is ever passed on — the “cascade” column is Jeff and its adapter alone. On the same seven, sending every decision to the 27B scores below Jeff; on guard, accuracy falls steadily from 98.0% toward the 27B’s 84.0% as more rows are escalated. Grounding is the one task where the 27B is better on its own and the cascade does route to it, 1.7% of the time.

So the result is stronger and narrower than the framing. Stronger: a task-tuned 0.8B adapter beat a 27B generalist outright on these tasks. Narrower: the test rows are synthetic — each records its source as the project’s own generator, jeff-finetunes/<task> — so they are held out by family but not by distribution, and the 27B ran with reasoning switched off, which the README’s caveats do say. Expect the adapters to be excellent at their nine jobs and treat the 27B as the generalist you keep for everything else, not a safety net under them.

On a CPU

The README’s speed table quotes 22 ms on an RTX PRO 6000, 28 ms on an M4 Max and 463 ms on a 32-thread CPU. On our four cores the replay above took a median of 2.2 seconds per decision, 5.8–6.3 s at the 95th percentile, with prompts of a few hundred tokens. Several questions in one request do not share work: the base took 906 ms for one question and 12.9 s for twenty, about 650 ms each. The process held 4.9 GB with the base and two adapters loaded.

The server takes one request at a time. Of eight sent together, one succeeded and seven returned 529 with Retry-After: 1 — the same overload code TypeSafe uses, so clients written for Jev will back off and retry rather than fail. It does not accept jev-latest, the model name those clients send by default, so pointing the official SDK at it needs the model set to jeff-latest as well as the base URL.

Without an adapter

The base is the zero-shot model, and on our standard probe it struggles. Is this a billing problem? separates billing from technical messages cleanly (0.96 and 0.95 against 0.07 and 0.11). Is it false that this is a billing problem? returned 0.58 and 0.76 for the billing messages — closer to agreeing than negating — and the positively worded inversion, which every other model we have probed handles, came back between 0.43 and 0.58 for all four. Use adapters for anything that matters, and keep questions short and positive on the base.

Verdict

The best-evidenced benchmark we have reviewed: seeds, row ids, model revisions and training-data hashes are published, and the two numbers we could reproduce came out the same. The adapters are the product. A v1.3 base is due within days and the current adapters will not carry over to it, so pin the revision you test. For many open models behind one daemon, see ollaya; for the other two projects called jeff, see Alurith’s CLI and logan-markewich’s server.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Oct 4, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.