Bespoke Nimble

How we reviewed this: Nimble’s 9B model needs Apple Silicon or an NVIDIA GPU with BF16, which our test machine doesn’t have. We read the code and documentation at commit d2387fc and ran its test suite on a CPU.

What it does

Bespoke Nimble, from Bespoke Labs, is “data, model, and recipe for an open Jev”. Nimble takes some text and a schema of questions, each a choice from a list you give or a true/false question (Score is also supported), and returns the chosen answer and the probability of each allowed answer in one step, without writing out reasoning. The model, Bespoke-Nimble-9B, is a LoRA fine-tune of Qwen3.5-9B.

The repository covers the whole pipeline:

• Data curation. Training examples come in contrastive pairs: two evidence sentences, where changing at most eight words in one of them flips the label. Separate model calls check each example, and code derives the labels from the checked facts.
• Training. LoRA on the answer tokens only, with a documented recipe.
• Evaluation. A held-out set of 324 examples, public benchmarks, and a comparison app.
• Serving. Local runners for MLX (Mac) and CUDA, and a GPU server on Modal.

The README compares Nimble with its base model and with Jev on the held-out set. Consistent with our policy, we don’t reproduce the Jev figures here.

Where the labels come from

The README says plainly: “we did not distill from Jev.” The curation guide and code back that up. New training examples are generated and checked with OpenAI’s GPT-5.6 models by default, with Claude as a supported alternative, and labels are computed in code from the checked facts. The pipeline also records Jev’s answers on the same examples, which are used for comparison. The README notes they could be used for distillation in future, but the current training uses labels “not derived from Jev”. All labels are synthetic and model-checked, with no human review, which the README states.

Setup

It needs Python 3.12 and either Apple Silicon (running Python natively for Metal) or an NVIDIA GPU with BF16 support. The weights alone take about 18 GB, and the one-time merge step needs extra RAM and disk. A download script checks that the model’s recorded prompt contract matches your checkout before running. Separate requirement files cover training, curation, MLX, CUDA evaluation, and Modal serving.

What we could check

• The tests mostly run on a CPU. 143 tests passed. The rest needed Apple’s MLX, dataset files that aren’t in the repository, or a separate environment. We installed two requirement sets together, which caused a version conflict the project avoids by keeping them apart.
• The serving API is Jev-shaped. The GPU server exposes /v1/systemone, /v1/models, and /v1/limits. It’s built on openjev-sglang at a pinned commit, credited in the docs.
• Its limits differ from Jev’s. Up to 26 options per question, 64 questions, and 2,048 prompt tokens. Unlike Jev, question IDs and option keys are part of the trained prompt, so name them meaningfully.
• Confidence isn’t calibration. The server describes its confidence as “entropy concentration, not calibrated accuracy”.

Permissions and data

• Local inference is private. Running the model needs no TypeSafe or generation API key, and nothing leaves your machine except the Hugging Face download.
• Curation calls paid APIs. Rebuilding the dataset sends prompts to OpenAI or Anthropic and, for the comparison, to Jev. Every request and response is saved, so the process can be replayed offline.
• Licensing is split. The model on Hugging Face is Apache-2.0, but the code repository has no license file, and the Modal server depends on openjev-sglang, which has none either.

Maintenance

Published on September 18 with a handful of large commits, thorough documentation (training, curation, datasets, benchmarks, serving), and two open issues. The README says it was built in one day and to “expect some rough edges”.

Verdict

Nimble is a remarkably complete open attempt at a Jev-style model: not just weights, but a careful, replayable way to build the data, with honest notes about what the labels are and aren’t. It’s most valuable to researchers and to teams with the hardware to run a 9B model. As a drop-in Jev replacement, mind the 26-option and 2,048-token limits, measure it on your own data, and resolve the code license before building on it.

For smaller models you can run on a laptop, see kev and jeff. For other open reproductions, see NanoJev and SemIf. Our Jev alternatives guide compares the approaches.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 19, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.