StartLux-Decision

How we reviewed this: we followed the README’s GGUF instructions on a CPU: the 0.8B Q8_0 file from Hugging Face, llama-server from llama.cpp, and the decision server that ships beside the file. We pointed the official Python SDK at it, sent malformed and out-of-range requests, and ran the repository’s own evaluation script against it on the three public JevBench tiers. Everything ran in Docker on two CPU cores with no GPU. We made no Jev calls. The README’s comparisons with Jev, its game results and its speed figures for H200 GPUs are the authors’ own, and we have not reproduced them.

What it is

A family of typed decision models: five dense sizes from 0.8B to 27B and a 35B-A3B mixture of experts. You send a state and typed questions, and each question comes back with a probability for every option. Nothing is generated: the model reads the option letters after one forward pass. Requests and responses use TypeSafe’s /v1/systemone format. The repository holds the inference package, a server, evaluation scripts, fine-tuning code and raw game logs. The weights are on Hugging Face and ModelScope, as original checkpoints and as BF16, Q8_0 and Q4_K_M GGUF files for llama.cpp.

The original checkpoints also read images and prompts up to 262,144 tokens. The GGUF path, which is the one a machine without a GPU can run, reads text only.

The number holds

The README’s GGUF table says the 0.8B Q8_0 file gets 177 of the 231 public JevBench items. We ran eval/suites.py from the repository, at its pinned benchmark revisions, against our CPU server:

tier	items	correct
easy	48	48
original	72	65
hard	111	64
total	231	177

That is 177, the same as the table. The scorer is JevBench’s own code, and the script scores any /v1/systemone server, so the result is easy to check. On two cores, the README’s three-question example took about 6 seconds. The longest hard-tier prompts, at 3,600 tokens, took 42 seconds for one question. llama.cpp read prompts at about 85 tokens a second for this model. The README’s speed figures, in tens of milliseconds, are on one H200 GPU.

Repeated requests were not bit-identical: the top probability of the same question moved in the third decimal place between runs (0.884, then 0.883).

Two different packages

The official SDK worked unchanged. It sent jev-latest, the server ignored the name and answered as sl-0.8b-q8, and every answer parsed. The values in it depend on which copy of the server code you run.

The README tells you to run gguf_server from inside the downloaded GGUF folder, so the package copied into that folder is what runs. In the 0.8B, 2B and 4B Q8_0 repositories that copy predates 3 October. For the README’s example it returned:

	probabilities	confidence returned	TypeSafe’s formula
team (choice)	0.884 / 0.003 / 0.113	0.884	0.826
severity (score)	0.032 / 0.374 / 0.593	0.593	0.341

docs/inference.md says confidence follows TypeSafe’s definitions and that the package returned the top probability “until 2026-10-03”. The GGUF folders still have that older package. Code that thresholds confidence will act differently here than against Jev. The same copy answers every bad request with a 500 and a Python repr: malformed JSON, an unknown question type, a missing questions, a score with twelve levels. It also answers a POST to any path, and GET /v1/models returns its health check.

Running the repository’s own gguf_server against the same folder gave what the docs describe: 0.824 and 0.341, 422 with a detail list, a request id and /v1/models. The original-weights repositories carry the current package. We reported it. Until it is fixed, run python -m startlux_decision.gguf_server from a checkout of this repository, with --model-dir pointing at the GGUF folder.

Memory

Our first run of the benchmark stopped partway through the hard tier. llama-server had been killed for using more than its 4 GB limit, serving a 0.81 GB file. Recent llama.cpp builds keep up to 8 GiB of prompt cache in RAM by default (--cache-ram 8192), and over the run its memory climbed to 7.6 GB. The decision server already asks llama.cpp not to cache prompts. With --cache-ram 0 it stayed at 467 MiB over 30 of the long hard-tier items. The README’s llama-server line does not set it.

Licence

The code is Apache-2.0. The weights are CC BY-NC 4.0: free for research and other non-commercial use, with attribution, and commercial use needs a separate licence from StartLux Labs. The README and every model folder’s NOTICE say so plainly. The weights are a modification of an Apache-2.0 model, whose notice is kept.

Verdict

One of the most reproducible open decision models we have tested. The published score for the smallest file came out to the item on a two-core CPU, with the authors’ own script and pinned data, and the official SDK needed no changes. Two things need fixing in the GGUF folders before the “clients written for Jev work unchanged” claim fully holds: the stale package, which reports a different confidence, and the README’s llama.cpp line on small machines. And check the licence before building a product on it. 386 stars.

For another open model in Jev’s format with TypeSafe’s confidence formulas, see Valen.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Oct 13, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.