Open Spark Jev

How we reviewed this: we ran both packages’ test suites on CPU, read the evaluation protocol against the code, and checked whether the leakage validator it relies on is tested. We did not run the model — it targets a GPU we do not have — and we make no claims about its speed or accuracy.

What it is

An independent local implementation “inspired by TypeSafe’s Jev and System One”, built for NVIDIA DGX Spark, releasing a model called spark-s1 in 1.7B and 4B sizes. The README’s own framing is careful: “it is not Jev, not a reproduction of TypeSafe’s proprietary system, and not affiliated with TypeSafe.”

It writes down the time it got it wrong

docs/EVALUATION.md is a protocol document, and this is the paragraph that made us take the rest of it seriously:

Two lessons from our own history: simulator corpora once leaked 97-99% of test states into training, and the training script’s validation split contained shuffled copies of training rows, which produced a badly fitted temperature (0.065). Any new data must be split by scenario group, never by row.

Publishing the magnitude of your own past contamination — 97 to 99 per cent — is not a thing projects do. It is the single most useful sentence in the document, because it tells a reader exactly which failure the current design is built to prevent, and it is the kind of detail that only ever surfaces when someone decides to write it down.

The rest is consistent with it:

• Four data sets with rules that never mix: training fits weights, calibration fits temperatures, development is burned once you look at it, and the locked release test is “read once; never tune, fit or look at rows afterwards”.
• On its own headline tool-call benchmark: “it is not a locked holdout and its numbers carry selection bias. Use it to show behaviour, not to claim generalisation.”
• A section titled “What a future locked release holdout should contain”, which is a list of everything the current evaluation is missing — written by people who did not build the model, hundreds of rows per family, labels double-checked by two reviewers, frozen and hashed outside the training repository.
• docs/API.md on calibration: “only the Choice temperature has been fitted … They are not task-specific and are not guaranteed to transfer.”

The document’s own subtitle is “what is still missing before a release can be called validated”, which is an unusual thing to advertise.

The check that would have caught it is tested

A protocol is only worth the code behind it, so we looked at the validator. split_isolation.py checks assign-time isolation properly: scenario ids reused across splits, template families shared between splits, entity pools shared between splits, and near-duplicate records by Jaccard shingles over visible text at a 0.92 threshold.

And it is under test. tests/unit/test_validation.py::test_split_isolation_rejects_reuse asserts it catches all three of scenario_id_reused, template_family_shared and entity_pool_shared.

That is the right relationship between a disclosed failure and the response to it: name the leak, build the check, test the check.

Both suites pass on CPU — 47 in the main package with the two GPU tests skipping, and 400 in datagen-pipeline.

A v5 release landed between our test run and publication, so we re-checked and found one failure: test_all_fifteen_packs_run_end_to_end_dry asserted exactly 15 packs, and the release that added “21 new Jev-style task packs … plus 19 code-only rule packs” took it to 34. Reported.

The reply is the reason to read that thread. The count is now derived from pack_names() rather than written down, and the test is renamed test_all_packs_run_end_to_end_dry so it cannot go stale the same way — but deriving it live surfaced a second, real defect the stale 15 had been masking: with the full pack list, two packs accepted zero rows even after the count matched. A hardcoded number had been hiding a generator bug, which is the argument against hardcoded numbers in one sentence. At 30ac6d8, datagen-pipeline is 400 passed and the main package 47 passed, 2 skipped.

What is not here, and now is

docs/EVALUATION.md said os-datagen “ships split_isolation.json as proof”. The validator that produces it was in the repository; the artifact was not, and datagen-pipeline/artifacts/ was empty — so the isolation claim was one you would have to regenerate on a GPU rather than read. We closed our review by saying that shipping the file would leave nothing else to ask for.

65da498 ships it, as docs/examples/split_isolation.example.json. We read it rather than take the line for it: ok: true, zero violations, and heldout_families covering all 34 packs, with the template-family, entity-pool and style-family maps beside them. The paragraph that links it is careful in the way the rest of this repository is careful — it says the file is a dry-run example of the report’s shape, “not evidence about the real generator’s output”, and that the real artifact stays a gitignored per-run file.

That is the right answer to the request. A checked-in example that overclaimed would have been worse than no file at all.

The labels themselves are the project’s strongest design decision, from ARCHITECTURE.md: 967 rows “whose labels come from code (policy engines, solvers, controlled worlds, a sandbox), not from an LLM judge”. A decision dataset labelled by a model is a dataset that inherits that model’s errors; labelling from executable policy is how you avoid it.

Verdict

We could not run the model, so treat this as a review of how the project reasons about evidence rather than of spark-s1’s quality.

On that axis it is among the best in this directory. It discloses a contamination incident with a number attached, writes the rule that prevents it, tests the code that enforces the rule, labels its data from code rather than from a judge, and tells you which of its own benchmarks you are not allowed to generalise from.

If you have a DGX Spark, this is worth your time. If you do not, read docs/EVALUATION.md anyway — it is a better statement of how to avoid fooling yourself with a decision benchmark than most of the benchmarks in this list.

We asked for one file at the end of our first review. It arrived the same day, with a paragraph explaining what it is not.

For the other two open reproductions with published protocols, see von and Dohnuts; for what a locked, recomputable benchmark looks like, Rizzo Flow.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 23, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.