Jev Benchmarks and Evaluation Tools

fstandhartinger/jevbench

Benchmark for typed decision models across several suites, with confidence cascades and committees reported separately.

sutro-sh/jev-align

CLI from Sutro that finds the examples a Jev function is least sure about, asks you to label them, and uses GEPA to improve the question.

xD3I/PZ_Optimization

Performance work on a game, notable here for the harness: the arithmetic and the noise floors are computed in code, and Jev is asked only for the verdict on what the numbers mean. No licence file.

danielgshea/jev-as-a-judge

Uses Jev through langchain-typesafe as the judge in an eval suite, asking typed quality questions instead of asking a larger model to grade. No licence file.

openlayer-ai/jevals

Agent evals and guardrails as typed questions instead of an LLM judge, packing every eval for a trace into one request. From Openlayer, with a mock backend so the whole library runs without a key.

apolinario/decision-index

Reproduction kit for a public leaderboard of typed decision engines: runs the public text suite against any /v1/systemone endpoint, locally or as one Hugging Face Job. Not affiliated with TypeSafe.

iammrduncan/inference-benchmarks

Compares LLM structured output with Jev on latency, cost, and judgment quality.

smkrv/jev-calibrate

Checks a Jev question against your own labelled examples and grades it: act on it, only sort by it, or rewrite it. Refuses to grade a question whose classes have too few examples, however good the numbers look.

deepdave98/jev-playground

Weekly go-to-market workflows built on Jev, such as lead triage and deal risk, each tested against Claude with its harness published. No licence file.

Zaious/jev-capability-atlas

A bilingual map of where Jev holds up and where it breaks, built from recorded API calls rather than a leaderboard. Its LICENSE is not a recognised open-source license.

AbdelStark/jev-benchmarks

Probability-aware evaluation: calibration, and how much work can be automated at a fixed error budget.

alexgreensh/eval-genius

Skill that tells a coding agent when a question needs an eval rather than a vibe, with an opt-in lane that routes the closed-label residue to a decision model and scripts for the cost and confidence arithmetic.

rlaope/jeval

Measures how well a classifier's confidence matches reality and turns the cost of a mistake into the threshold where a human should take over.

erendikmenn/jev-rag-benchmark

Reproducible experiments on whether reranking with Jev improves a small RAG system, on a locked Turkish dataset, with quality, latency and cost reported together.

QuicqDev/Jev-vs-ML

Compares a typed decision model with classical classification pipelines across eight datasets, with a published protocol and an interactive report.

TrustifAI/typed_evals

Python toolkit that puts evaluation of LLM, RAG and agent output as typed questions, with optional calibration against human labels.

JevBench/jevbench

Tests whether a decision model's probabilities fit together, using 50 laws of probability and choice and no gold labels. Unrelated to the JevBench above.

keta1930/what-the-jev

Reproducible experiments and reports on what Jev can and cannot do, run through OpenRouter: knowledge benchmarks, bias, skill routing and price prediction.

kunko-ai-labs/judge-audit

Pre-registered calibration audits of AI judges, comparing Jev's native probabilities with LLMs' verbalised confidence and vote shares, caveats stated beside every result.

ishida-lab/ProbabilityBench

Synthetic text worlds with exact ground-truth probabilities, for scoring how well decision models such as Jev are calibrated, through OpenRouter. With a paper.

beerandcodeteam/workshop-jev

Laravel workshop app with a live chat whose messages are moderated by an LLM agent, Jev, a local Laya model or OpenAI Moderation, compared side by side. Portuguese.

akitaonrails/jev-docs-benchmarks

Research notes and a benchmark comparing hosted Jev with other hosted and local decision models on following supplied rules, probability behaviour and operating cost, with frozen inputs and raw response archives. No licence.

JuneYaooo/jev-healthcare-lab

Compares Jev with DeepSeek on healthcare judgement tasks, from triage urgency to contraindications and trial referral, with the task sets, scoring and raw results published. Exam cases, not clinical validation. Chinese and English.

abhixhek/jevcal

Picks the confidence threshold that meets your accuracy target on your own data, and fails CI when a model update breaks it.

anisselbd/jev-phishing-bench

Reproducible benchmark that runs Jev and Claude Haiku 4.5 over the same 2,000 phishing emails and compares accuracy, calibration, stability across repeated passes, latency and cost, with every step scripted.

FirasSX914/Janus

Measures Jev's calibration and confidence-based routing on Banking77 and Web of Science.