Contrastive Language Models

How we reviewed this: we did not run it, and that is a choice rather than an omission. CLM-8B is a 75 MB decision head on top of Qwen3-8B embeddings served by vLLM, which needs a GPU this machine does not have; a stand-in encoder would have produced answers about nothing. So this is a code read plus the check a shipped benchmark deserves: recomputing its published comparison from the per-seed records in the same files. No Jev calls, and every figure below is the authors’ own measurement, not ours.

What it is

A “System One” model trained with a contrastive objective that embeds states and actions separately and scores one against the other — so both sides cache independently, which is where the speed claim comes from. Pre-trained on 60M Nemotron Q&A pairs, mid-trained on synthetic hard negatives, post-trained on agentic trajectories, and served behind POST /v1/systemone with /v1/rank and /v1/models beside it. Apache-2.0, pip install contrastive-lm, weights and the reference head on Hugging Face.

Setup, and what we could not do

Two processes: vllm serve Qwen/Qwen3-8B --runner pooling for the encoder, then clm-serve for the decision head. The encoder is the expensive half and the one we could not stand up. Worth knowing before you try: --max-model-len 2048 on the encoder and --max-tokens 2048 on the server must be raised together, and the default truncates a longer state at the embedder — truncate_prompt_tokens is set on every request, so a 4,000-token state is silently shortened rather than refused. The README says states are truncated; it does not say the server will not tell you when it happens.

What its own run says

examples/t_rex/ ships a real-time game benchmark with two result files — one driving the hosted Jev API, one driving CLM locally, five seeds each. We recomputed every aggregate in both summaries from the per-seed records:

seeds · survived · deaths · mean_best_score · mean_decisions · mean_agreement_with_planner
latency_ms_p50_median · model_ms_p50_median · errors · answers_discarded      10 of 11 exact, both files

That is a good result, and the interesting part is what the numbers then say. Both systems finished all five seeds with no deaths and the same mean best score, 697. Underneath that tie, the authors’ own records show CLM making about three times as many decisions (3,341.8 against 1,119.0), agreeing with the planner far less often (0.658 against 0.987), and needing far more safety interventions from the harness. Their latency figures are 16.5 ms against 149.8 ms at the median.

So the honest reading of their own artifact is narrower and more interesting than “on par with Jev with up to 9× lower latency”: in this game, at a tenth of the latency, it takes many more shots, agrees with the planner much less, and the harness catches the difference. Both end up at the same score. That is a real result about a fast decision model in a closed loop, and the repository has the records to support it.

The one number that does not recompute

shield_interventions in the summary is not the sum of the per-seed field with the same name:

file	per-seed shield_interventions summed	summary shield_interventions
jev_realtime.json	0	28
clm_realtime.json	364	4,883

It is the sum of three counters — shield_interventions + arrival_saves + emergency_saves — which examples/t_rex/run.py:131 does explicitly, while line 82 copies the pilot’s own single counter per seed. Same name, two meanings, one nesting level apart. It matters because this is the most dramatic number in the comparison: read per-seed, CLM’s shield fires 364 times; read from the summary, 4,883. Renaming the summary field safety_interventions_total would settle it.

Permissions and data

Self-hosted throughout: the decision head talks to your own encoder at CLM_EMB_URL, the client to your own server at CLM_BASE_URL, and CLM_API_KEY is checked only if you set one. Nothing leaves the machine unless you point the encoder somewhere else. The Jev half of their benchmark, by contrast, went to api.typesafe.ai — the endpoint is recorded in the result file, which is the right way to publish a comparison.

Maintenance

Apache-2.0, active, with training, preprocessing and evaluation directories, a serving script, and a documented pip install. The claims beyond this game — parity across computer-use and tool-calling, Terminal-Bench 2.1 at 87.6%, DeepSWE at 81.6% — are the authors’, and the repository does not ship those runs’ records the way it ships this one’s. Take them as reported, and the t-rex run as checkable.

Verdict

The most prominent open decision model in this directory right now, and the rare one whose headline comparison can be audited from the files beside it. If you have a GPU for the encoder, it is the serious candidate for a closed loop where latency dominates.

Read examples/t_rex/results/ before the README: it is more informative, and it is the part we could verify. Then decide whether three times the decisions at a tenth of the latency is the trade your loop wants. For the same shape from published weights on smaller hardware, see JevK5 and decider; for the board these systems are ranked on, JevBench.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 27, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.