Laya

How we reviewed this: we ran its lint and the test files that need no weights, then pulled both shipped checkpoints and drove the Router on CPU with --network none across eight languages. We reconciled its published calibration figures against the JSON it commits. We made no Jev calls — this is an open model and makes none.

What it is

A non-autoregressive decision engine: typed choice, score and noul questions answered in one forward pass, with no text generated. Three checkpoints — a 421M English model on ModernBERT-large, a 322M multilingual one on mmBERT-base, and a fine-tuned typed-decisions variant — and a Router that detects the script and language and dispatches per request.

Apache-2.0. Fourteen and a half thousand stars and more than a thousand forks, four days after the repository was created.

A note on our own listing: we ignored this project in a bulk triage on 2026-09-18 and were wrong to. It is the thing two projects we already listed are built on top of. This review exists because that call was reversed.

It works, in seven languages, on a laptop CPU

One complaint — “I was charged twice and nobody has replied for three days” — translated into eight languages, one choice question, --network none:

language	script	routed to	answer	confidence
English	latin	english	negative	0.816
Chinese	han	multilingual	negative	0.973
Hindi	devanagari	multilingual	negative	0.990
Arabic	arabic	multilingual	negative	0.965
Japanese	kana	multilingual	negative	0.987
Russian	cyrillic	multilingual	negative	0.989
Spanish	latin	english	negative	0.535
English, positive control	latin	english	positive	0.622

Every answer is right, including the control. Script detection and dispatch behaved exactly as described on every non-Latin script, and a warm call took 49–136 ms on CPU — the project’s own 33 ms figure is on a T4 and is its measurement, not ours.

Two things we reported, both now fixed

The Router reloads a checkpoint on every script switch. max_loaded defaults to 1, so alternating languages evicts and reloads. Same machine, same inputs, both checkpoints already warm:

max_loaded=1    en 21235 ms · zh 21144 ms · en 20677 ms · zh 23306 ms
max_loaded=2    en   130 ms · zh    49 ms · en   136 ms · zh    49 ms

A 150–400× difference on mixed traffic, from one constructor argument. For a component whose headline is per-request routing, 1 is the wrong default — and the fix is already in the API.

Fixed in 0.3.7 (PR #180, merged 2026-09-23), with a regression check that counts rebuilds over an alternating workload. We re-checked against the published package rather than the branch: inspect.signature(Router.__init__) on laya 0.3.7 from PyPI gives max_loaded: int = 2, and the alternating workload no longer reloads anything.

max_loaded=2 (default)   en 169 ms · zh  52 ms · en  145 ms · zh   52 ms   loaded=['english', 'multilingual']
max_loaded=1 (opt-in)    en 982 ms · zh 3381 ms · en 1079 ms · zh 3387 ms   loaded=['multilingual']

That is this box with a warm page cache, so the penalty is smaller than the 21-second reloads we measured a day earlier on cold storage — but it is the same failure, and it is now something you have to ask for rather than the default.

Spanish and Italian routed to the English checkpoint — fixed. detect_language returned is_english: True for some Latin-script Romance text, which mattered because the project’s own limits say laya “collapses outside English”: the same complaint scored 0.96–0.99 through the multilingual checkpoint and 0.535 through the English one.

Both reported. This half was fixed within hours, by a third contributor rather than by us, in a way that found more than we had: the Romance stop-word lists held mostly accented words, so a state whose accents had been stripped by a mail client or a ticket system matched one function word or none. The maintainer merged it after checking it on real data — on MASSIVE, plain-ASCII Italian routed to multilingual went from 39% to 80% and French from 54% to 76%, with no English prose moving across 20,231 states.

We re-ran our own cases at c752770, and again on the published 0.3.7, model-free — detect_language() needs no weights:

state	language	is_english	routes to
Spanish, accented and accent-stripped	es	False	multilingual
Italian, accented and accent-stripped	it	False	multilingual
French, accent-stripped	fr	False	multilingual
Portuguese, accent-stripped	pt	False	multilingual
English control	en	True	english
English, too short to identify	None	True	english

Both spellings of each state, accented and accent-stripped, give the same answer. The last row is the deliberate part: an unidentified state with no non-English letters still goes to English, and the comment beside it explains why the opposite default was worse.

A related routing thread (#177) is open, on a Portuguese stop-word list that contains com; we did not reproduce it with our own inputs and have no finding of our own there.

The part that earns the listing

BENCHMARKS.md ends with a section called “Limits, stated plainly”, and it does what the title says:

• Near chance on typed-decisions zero-shot — the 0.766 belongs to the fine-tuned checkpoint, on that benchmark’s own training split.
• Moderation does not hold up on held-out data (0.530, macro-F1 0.400).
• Both checkpoints ship over-confident. Fit temperatures on your own data.
• Ordinal score is the weakest primitive.
• laya collapses outside English; laya-multilingual is weaker on English. Route.

The first line is the one to notice: it tells you a headline figure came from the benchmark’s own training split. Very few projects write that sentence about their own best number.

The benchmark header is equally careful about the comparison everyone will look for: “Jev figures are third-party published, never measured here — no TypeSafe API access — so sample sizes and prompts differ; treat them as indicative.” We do not republish other projects’ Jev comparisons, so the figures are not reproduced here; what we can say is that they are labelled as not the author’s measurements, and that the tables include rows where Laya loses badly rather than only the ones it wins.

And it commits the evidence. research/results/t4_colab_benchmark.json reconciles exactly to the calibration table:

	published	in the JSON
laya ECE, shipped → refit	0.466 → 0.081	0.4656 → 0.0812
laya-multilingual	0.314 → 0.106	0.3135 → 0.1059

The same file carries a caveats block stating that AG News and BoolQ were in the training mix — “retention, not generalisation”. The README’s own table marks those rows “in training mix”; the headline table in BENCHMARKS.md does not, which is the one place it would matter most.

One of the three result files the benchmark names, app_benchmark.json, has never been committed — already reported by someone else.

The library polices its own weights

Loading a checkpoint printed this, unasked:

laya: this checkpoint ships temperatures outside [0.5, 5] which would distort confidence;
clamping choice:11+=0.1006. Treat confidence from the affected buckets as uncalibrated.

A library that inspects its own published weights, finds a calibration parameter out of range, clamps it, and tells you which buckets not to trust is doing something we have not seen elsewhere in this directory.

The noul question

There is an open report that noul returns the negative label regardless of the state. We could not reproduce that as stated — it does discriminate — but the asymmetry behind it is real:

	positive input	negative input
noul	0.6164	0.0000, confidence 1.0
choice, labels positive/negative	positive @ 0.842	negative @ 0.963
choice, labels yes/no	yes @ 0.465	no @ 0.891

Near-certainty on “no” and near-coin-flip on “yes” for equally unambiguous text, and the confidence moves with the label words rather than the primitive. We added our numbers to that thread.

The thread has since found the mechanism, and we confirmed it on 0.3.7. A noul renders its two options by reading crit.get("false") and crit.get("true") and nothing else, while the validator only checks that criteria is a dict. So a noul written {"yes": …, "no": …} — the spelling most people reach for — is neither rejected nor used: the descriptions are dropped and the defaults are rendered. Twelve asks on three positive and three negative reviews, both checkpoints: “no criteria” and {"yes","no"} are identical to four decimals on every one.

It is not a cosmetic loss. Putting the same descriptions under the keys the renderer reads changes the answer on the English checkpoint:

question	pos 1	pos 2	pos 3	neg 1	neg 2	neg 3
noul, no criteria	0.5914	0.4625	0.6364	0.0001	0.0073	0.0111
noul, criteria {"yes","no"}	0.5914	0.4625	0.6364	0.0001	0.0073	0.0111
noul, criteria {"true","false"}	0.8486	0.7881	0.8723	0.0000	0.0007	0.0014

Two of three positives become three of three, from the key spelling alone. Uppercase is fine (TRUE/FALSE are lowercased); anything else is dropped.

So, until the checkpoint or the API changes: spell a noul’s criteria true/false or not at all, and if the answers still look stuck, use the two-option choice the 0.3.7 README now recommends — {"A": "yes, …", "B": "no, …"} answered 6 of 6 correctly on the English checkpoint for us and 5 of 6 on the multilingual one. Do not threshold on a noul confidence.

Engineering

Apache-2.0, Python 3.10+. ruff with the project’s own CI rule set is clean, compileall passes, and the five test files that need no weights all pass. The rest need a checkpoint, which is fair.

The issue tracker is at 172 and moving fast, with substantive reports from outside contributors — several of which are about exactly the kind of thing this review found. That is a healthy sign four days in, and also a warning that the code under you is moving.

Verdict

The most interesting open decision model in this directory, and the honest one. It answered a real question correctly in seven languages on a CPU with no network, and it told us where it fails before we could find out ourselves.

Route deliberately — on 0.3.7 the defaults are finally the right ones, and if you use more than two checkpoints, set max_loaded to match. Fit your own temperatures; the project says twice that it ships over-confident. Prefer choice with descriptive labels while the noul asymmetry is open.

Then read BENCHMARKS.md to the end. The last six bullets are worth more than the tables above them.

For the two projects built on these checkpoints, see Laya for Node and DeepOpen; for a smaller open model read out of logits instead, JEV-CPU.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 23, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.