jev-eval-mcp

How we reviewed this: we built it in Docker, ran its tests, and drove all three tools against a local stand-in, including a full evaluation sweep over 16 labelled cases. We made no Jev calls.

What it does

jev-eval-mcp, by BYK, is an MCP server with three tools: ask one question and see the whole distribution; map a question over a list of items; and evaluate a question against labelled data, producing a threshold sweep with precision, recall, AUC, Brier score, calibration error and the worst misses.

That third tool is the point, and it’s the right point: the gap between “this probability seems high” and “at 0.8 I catch 91% and wrongly flag 4%” is where most teams get stuck.

Two things to fix first

save_path is unvalidated. Both the map and evaluate tools write their output to whatever path the caller passes, with no restriction — and in MCP the caller is a model. An agent that decides to save results to ~/.ssh/config will do exactly that. A workspace-relative check would close it.

The by-hand script ignores your base URL. The documented “drive one tool manually” script builds a child environment containing only the API key and PATH, dropping TYPESAFE_BASE_URL and the model override — so it always calls the real API regardless of what you configured. Both are reported upstream.

Also worth knowing: the only test file covers the metrics. The tools, the client and the schemas have none.

Verdict

The idea deserves copying — an eval tool sitting in the agent’s own toolbox, so choosing a threshold is a thing you do rather than a thing you postpone. Fix the write path before letting an agent near it.

For calibration as a standalone workflow, see jevcal.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 20, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.