solar-mini4-jev

How we reviewed this: we read the request path, ran the real server in python:3.12-slim against a stand-in upstream that logged every header it received, drove it with the official typesafe-sdk 0.7.0, and recomputed every number in the README from the gold labels and raw results committed in bench/. We made no Jev calls and no Upstage calls. The head-to-head figures below are the author’s; we verified that they follow from the author’s data, which is a different claim from verifying the models.

What it is

A 1,200-line wrapper that accepts Jev’s POST /v1/systemone body, turns the questions into a JSON schema, asks Upstage’s Solar Mini4 to fill it in, and returns Jev’s answer shape. Bring your own Upstage key. There is a hosted instance on Vercel and a local server you can run yourself.

The official SDK drove it with nothing but a base URL:

answered: type='noul' noul=0.81

The benchmark is the best-evidenced in this category

The README’s headline sentence is: “Out of 447 scored answer fields, Solar Mini4 missed 7 and Jev missed 26. On the 31 fields where the two disagree, Solar is right on 25 and Jev on 6.”

bench/ holds the generated cases, the gold labels and the raw per-case results, so we recomputed that from scratch — reading the gold file and the results JSONL and applying the project’s own scoring rule (a noul agrees within 0.25 and on the same side of 0.5; a choice must match exactly; a score within 1):

	fields	Solar misses	Jev misses	errors
our recomputation	447	7	26	1
the README	447	7	26	—

And the split:

  both right        415
  only solar right   25
  only jev right      6
  neither             1
  they disagree on   31

Exactly the sentence. Every figure, to the unit. That is the first time this month a published head-to-head has reproduced from its own artifacts without a single adjustment, and it is worth more than the numbers themselves.

And the gold labels are not Jev’s answers. The README is explicit: “Jev is not the gold standard. Jev is a peer model under comparison.” The reference labels come from a third-party judge rubric. Half of this category grades itself against the very model it is competing with, which can only ever measure imitation; this one did the harder and correct thing.

One loose thread in the provenance. The README names gold_grok46_judge_test400.json as the official reference and then quotes the numbers from the reasoning_none run (447 fields, 7 misses), while summary_grok46_judge_official.json records a later re-run at 448 fields and 6. Both use the same gold, both reproduce exactly from their own results file, and the difference is one case that errored. It is a sentence to tighten, not a result to doubt.

The prompt knows the scenarios

engine.py’s system prompt is a 33-line hand-written calibration rubric that names specific situations and the confidence each deserves:

Immediate protective action: HIGH (~0.9) only for truly catastrophic signals (active flames+injury life-safety, evacuation wildfire red, oxygen confined-space emergency, ransomware EHR with outage, structural collapse). Mid-severity (freezer minutes, RH vault, soft-tissue SIU, anti-cheat injector, CO2 drift, cyber CAT desk open, bridge strain without collapse, stadium vibration, caseworker breach, cold-room hold) → MID (~0.65–0.78), not 0.95.

There is a proper train/test split — cases_train400.json and cases_test400.json, with the tuning summaries named after the train split — so the prompt was not fitted to the reported items. But both splits come from the same generator, and 12 of the 13 distinctive scenario terms in that rubric appear in the test cases as well as the training ones.

So the split is at the item level, not the distribution level, and the rubric encodes the distribution. The peer model in the comparison receives the plain question with no rubric at all. That does not make the result wrong — the prompt is part of the system you would deploy, and saying so is fair — but “a prompt tuned to this family of scenarios, against a model that has never seen the family” is the sentence a reader needs, and it is not in the README.

What happens to your API key

This is the finding.

server.py accepts the upstream key from any of three headers, and the third is the one the official SDK uses:

if authorization:
    raw = authorization.strip()
    if raw.lower().startswith("bearer "):
        token = raw[7:].strip()
        if token:
            return token

engine.py then forwards it verbatim:

headers={"Authorization": f"Bearer {_key(api_key)}", ...}

to https://api.upstage.ai/v1/chat/completions. Nothing validates the key’s shape — any non-empty string is accepted and sent.

A caller doing exactly what the README invites — reuse the official client, change the base URL — sends Authorization: Bearer <their TypeSafe key>. We ran it, with the upstream replaced by a server of ours that logs what arrives:

[0] POST /v1/chat/completions
    Authorization: Bearer sk-typesafe-CANARY-DO-NOT-LEAK
    model: solar-mini4  messages: 2

The TypeSafe-shaped credential went straight to the upstream. On the hosted Vercel instance it makes that trip through a third party’s deployment first.

The mitigation is a few characters: Upstage keys have a recognisable prefix, so refusing a bearer token that is not one — with a message saying which key is wanted — turns a silent cross-vendor leak into a 401. We have reported it.

Two smaller things

There is no GET /v1/models. The server exposes /, /health and /v1/systemone. client.models.list() returns 404 — the same gap as OpenThai-SystemOne. Anything that enumerates models before calling will trip on it.

UPSTAGE_URL is a hardcoded constant, while the model name, the reasoning effort and the key all read from the environment beside it. So you cannot point this at an Azure-hosted Solar, a gateway, a logging proxy or a test double — and you cannot exercise it at all without sending a real key to Upstage. We needed one patched line to review it, reverted and diffed afterwards. Making that constant read os.environ.get("UPSTAGE_URL") like its three neighbours would cost nothing.

Verdict

The benchmark discipline here is the best we have seen in this category: a third-party judge instead of a rival’s answers as truth, a real train/test split, every artifact committed, and every published number reproducing to the unit. If you are building a comparison of your own, read bench/ before you read anything else.

Then fix the key handling before you point a client at it, and pass the key in X-Upstage-Api-Key rather than Authorization so that a mistake is a 401 rather than a credential in someone else’s logs.

For the same shape on a model you host yourself, see JevBERT and the rest of the open models category.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 22, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.