Jev Plays StarCraft

How we reviewed this: we ran its test suite offline in Docker and independently recomputed the statistics in its published decision log. We could not run the game: the setup script clones a third-party emulator and needs the game package.

What it does

tsai-sc, by phyous, plays the first combat mission of the StarCraft shareware campaign. The 1998 executable runs in an x86 emulator inside a headless browser; a small bridge speaks to it over the debugging protocol.

The crucial detail is that game state comes from reading the emulated machine’s memory, not from screenshots — a build signature is checked, then the unit pool is walked at a fixed stride. Jev answers a Choice about what kind of thing to do next and a second about which concrete command, from candidates that code assembled: which four enemies may be focused, where to retreat, where to scout, what to build. 282 tests pass offline, none needing a key.

Recomputing its claims

Its decision log is published, and we recomputed from it rather than repeating the README. The interesting numbers are the ones the README doesn’t highlight:

• One request every ~2.5 seconds of wall clock, about 22,000 input tokens each, with up to 94 candidate commands in a single question.
• Roughly 44% of the wall clock had the game paused while it observed and decided. A real-time comparison this is not, and the repository’s own reference notes say TypeSafe’s Doom demo runs about 25 times more often.
• Jev chose “do nothing” on 116 of 421 decisions, and about 9% of issued commands were never acknowledged.

What holds up: every recorded choice is the argmax of its own probabilities, every executed action matches the candidate that was selected, and the published victory screenshot’s hash matches the manifest. Nothing was substituted after the fact.

What doesn’t: the README says a person must inspect the original screen before a run is presented as a victory. The repository’s own review file records the method as an AI inspecting the retained screenshot. That’s the one claim we’d want changed, because it is the step the project itself identifies as the human check.

Two smaller things: the files the manifest hashes are gitignored, so a fresh clone can verify nothing until you download a 399 MB release asset; and the verifier compares hashes recorded by the same run that produced them, which proves nothing was edited afterwards rather than where it came from.

The mission is also the easiest one — the shareware campaign’s first combat map, with a generous starting position and no expanding opponent. The README calls it a bounded experiment rather than a benchmark, which is the right framing.

What it sends

A structured summary: squads, visible enemies, resources, map coordinates. We scanned the 44 MB published log and found no screenshots, no base64 blobs, and no credentials of any kind. Key handling is the strictest in this round: an environment file must be a regular file, owned by you, with no group or other permissions, opened without following symlinks.

Local caveat: the bridge and the debugging port are unauthenticated on loopback, so any local process can drive the emulator.

Verdict

An honest demo, and the development notes — which record failures it had every reason to hide — are the best thing in the repository. Fix the line about who did the visual check, and this is a model for how to publish a demo: logs included, claims recomputable.

For other game demos, see TypeSafe Mario and jev-drone.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 20, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.