choosekit

How we reviewed this: we ran it against llama.cpp b11042 serving LFM2.5-1.2B-Q8_0 in Docker, on a 4-core i3-9100T with no GPU, machine idle. We made no Jev calls, and per our policy we do not republish the project’s own Jev comparison figures — the numbers below are ours.

What it does

Point it at a llama.cpp server you are already running, give it a context, a question and a list of options, and it returns a decision with a full probability distribution. It gets there by walking a token tree and forcing the decode — logit_bias: [[targetTokenId, 1_000_000_000]] — so it reads the conditional log-probability of each option rather than asking the model to write one.

It is not a Jev replacement in the API sense and does not claim to be. There is no HTTP server, no /v1/systemone, and the unit is one question, not a state with a question set. The README says plainly: “choosekit is an independent project with no affiliation to TypeSafe or Jev.” A choosekit-mcp subpackage exposes the same capability as a stdio MCP tool, listening on no port.

The engineering is tight. package.json has no dependencies field at all, only typescript pinned exactly as a dev dependency, and tests/package.test.mjs asserts both that there are no runtime dependencies and that importing the public entry points calls no fetch and logs nothing. npm test gives 138 passed, 0 failed, 0 skipped at 1014958 — it was 94 two days ago, and the suite has grown with the backends.

Best of all, it validates its own backend rather than degrading. src/llama-cpp.ts:141:

if (isRecord(settings)
  && (settings.post_sampling_probs === true || settings.backend_sampling === true)) {
  throw new ScoringError("llama.cpp applied incompatible post-sampling probability settings.");
}

It sends post_sampling_probs: false, backend_sampling: false and then checks the server honoured them. A tool whose entire output is a probability distribution refusing to produce one from post-sampling numbers is precisely right. The cost is rigidity: swap in vLLM or Ollama’s OpenAI endpoint and it stops working rather than quietly giving you worse answers.

What we measured

One question, three options, twenty timed calls after three warm-ups, serial, machine idle:

median	233 ms
p25 / p75	222 / 245 ms
p95	316 ms
node RSS	70 MB

The steadiest numbers in this group — p75/p25 is 1.10 — because cache_prompt: true is on and 64% of prompt tokens came back cached in our runs.

Be careful comparing it to anything: that is one question, where the other local projects we measured answer three per request. Scaled naively it is around 700 ms for three, but choosekit makes three separate calls with five to seven HTTP round trips each.

We also ran the bundled SemIf harness against LFM2.5-1.2B over the first 40 rows:

byPartition.familiar_mechanism_new_situation: rows 40, accuracy 0.425, balancedAccuracy 0.420
byFamily.evidence_interpretation: rows 24, accuracy 0.375
byFamily.rule_application:        rows 16, accuracy 0.500

17 of 40. The harness works out of the box, and that number is a statement about a 1.2B model, not about choosekit. A larger model would do better; the library is the same either way.

A worked example of what that means. Given a plainly technical support ticket, it returned:

choice  {"choice":"billing","distribution":{"billing":0.9240,"technical":0.0263,"sales":0.0498}}

Wrong, at 0.92. The library did its job — that is what the model believed. src/types.ts:27 is upfront: “Shannon entropy in nats, not a probability of correctness.”

Its benchmark could name the wrong model; now it checks

benchmarks/run-semif.mjs defaulted --model to qwen3.8-27b-text-64k, and llama.cpp does not reject an unknown model name. We started a server holding only LFM2.5-1.2B, sent a request naming qwen3.8-27b-text-64k, and got a 200 with the loaded model’s answer. So a benchmark run without --model labelled every row with a model that may not have been involved, and nothing warned you.

Reported, and the author asked for a patch; ours was merged the same evening. Re-run at 1014958 against a real llama.cpp holding one model:

run	result
default --model	does not list "qwen3.8-27b-text-64k". Available models: "qwen0.5b-q8", exit 1, no output file
--skip-model-check	runs, warns recording unverified model ID, writes modelChecked: false
--model qwen0.5b-q8	Model ID confirmed in the server catalog, writes modelChecked: true

The report also records the dataset revision and a SHA-256 of the input file, so a result file now says what produced it. That still leaves one thing. benchmarks/README.md:41:

Benchmark result files are ignored because they can contain environment-specific timing and provider metadata.

The reasoning is sound, but the effect is that the comparison table on the README’s front page has no committed raw data behind it, and reproducing it needs an RTX 4090 and a paid API key. We could not audit it, and we do not republish other projects’ Jev comparisons in any case. Our own 233 ms and 42.5% above are unrelated to that table. benchmarks/results/ is still in .gitignore at 1014958, so this one stands.

Verdict

Apache-2.0, 27 commits, two days old, CI green on master and every tag. The benchmarks/data/semif-authored144.jsonl dataset is committed byte-for-byte with its MIT licence file and the upstream commit hash, which is exactly how to vendor a benchmark.

Two limits worth knowing before you build on it: labels mode caps at 26 options and replaces your keys with A/B/C in the prompt, so option descriptions have to stand alone; and the MCP subpackage adds a floating zod ^4.6.4, unlike the zero-dependency core.

It is moving quickly. Since we first ran it, an OpenRouter backend, a SuperGPQA benchmark and an Ollama backend have landed, and llama.cpp is no longer the only thing it talks to — the default base URL is now Ollama’s 127.0.0.1:11434. The zero-dependency core has survived all three.

We’ve reported the benchmark’s model attribution and, at the maintainer’s invitation, sent a pull request that checks the requested model against the server’s own list before a run. For the same job with a batched API shape, see jev_local; for a library that answers a whole question set in one forward pass, open-alternative-jev.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 22, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.