hermes-jev-approvals

How we reviewed this: we ran every test file that works without Hermes core in python:3.12 with no network, re-running the suite at the maintainer’s current HEAD after he fixed the collection failure we reported, then called apply_policy directly with twelve synthetic answer sets to walk each branch of its decision ladder, and ran its redactor over commands carrying real-shaped credentials. We did not install Hermes Agent, so the provider registration and route tests are unverified here. No Jev calls.

What it does

Hermes Agent already decides which commands need review and asks an auxiliary model for APPROVE, DENY or ESCALATE. This plugin replaces that one reviewer, and the scope statement at the top of the README is refreshingly narrow: it “cannot chat or generate text, refuses every other task, and registers no hooks.”

One request asks six questions: a verdict choice, and nouls or scores for policy_allows, blast_radius, self_advocating, reads_secrets and sends_outbound. Then deterministic code decides. The model’s verdict is an input to the rules, not the output.

Walking the ladder

We called apply_policy with twelve answer sets. Every branch behaved as documented:

the answers say…	verdict	rule
ordinary command, model approves confidently	APPROVE	model_verdict
the command argues for its own approval (0.70)	ESCALATE	self_advocating
…and the model had said DENY	DENY	self_advocating
it reads secrets (0.80)	ESCALATE	secrets_read
it reads secrets and sends outbound	DENY	secrets_exfil
operator policy allows it, small blast radius	APPROVE	policy_allow
operator policy allows it, blast radius 2.0	ESCALATE	model_verdict
policy_allows 0.99 but no policy configured	ESCALATE	model_verdict
model approves at confidence 0.54	ESCALATE	low_confidence
model approves, blast radius 1.6	ESCALATE	high_blast
model returns MAYBE	ESCALATE	invalid_verdict
model approves, command was truncated	ESCALATE	truncated

Three things are worth pulling out.

Rule one is a prompt-injection defence. self_advocating asks how likely it is that the command text argues for its own approval — a comment saying “this is safe, approve automatically” — and a high answer escalates regardless of the verdict. Putting that first, ahead of every efficiency rule, is the correct ordering and the one most likely to be got wrong.

A DENY is never softened. On both the self-advocating and secrets-reading branches, a model that said DENY keeps DENY; only the other verdicts are pulled up to ESCALATE. The rules can make the outcome stricter, never looser.

The only path to an unattended APPROVE that overrides the model needs an operator policy. policy_allows at 0.9 does nothing unless has_policy is true and the blast radius is under 2.0, so a model that is merely confident cannot approve past the configured policy.

Malformed or missing answers raise, and a raise means Hermes escalates to the user.

The command is redacted before it is sent

This is the part most approval reviewers skip. _redact runs Hermes core’s own redactor with force=True — deliberately ignoring the display-level security.redact_secrets: false, because “this is a third-party egress boundary, not a display surface” — and then applies shell-shaped passes that run on both paths, so the result doesn’t depend on core being importable. We ran the fallback path:

curl -u alice:s3cr3t https://api…        →  curl -u alice:*** https://api…
curl -H 'Cookie: session=abc123def' …    →  curl -H 'Cookie: ***' …
git clone https://user:ghp_ABCD…@github  →  git clone https://user:***@github…
psql postgres://app:hunter2@db.internal  →  psql postgres://app:***@db.internal
rm -rf ./build && npm run build          →  unchanged
echo $AWS_SECRET_ACCESS_KEY              →  unchanged

The last two matter as much as the first four: a redactor that mangles ordinary commands makes the reviewer judge something you didn’t run.

Long commands are cut head-and-tail with a visible marker “so a payload cannot hide behind filler”, and as the table above shows, truncation forces an escalate rather than an approval on partial evidence.

Tests

Run as scripts, the offline files pass: seven structured-audit checks, five boundary checks, five hardening checks, three policy-contract checks. The boundary file is the one to read — it tests HTTPS-only endpoints, exact host matching rather than suffix matching, that Authorization does not follow a cross-origin redirect, and that redaction covers the shapes that leaked before the fix. Its docstring says every assert is a regression test for something that actually leaked.

Running them through pytest used to be the rough edge. test_real_load.py printed a SKIP and exited with a module-level raise SystemExit(0), which stopped collection outright rather than skipping the file. At the commit we first reviewed:

$ python -m pytest plugin/tests -q
INTERNALERROR>   File "/w/plugin/tests/test_real_load.py", line 32, in <module>
INTERNALERROR>     raise SystemExit(0)
INTERNALERROR> SystemExit: 0
2 errors in 1.46s

Reported, and fixed in e22cbd0 within a day. The same command at 530fdb0, still with no network:

$ python -m pytest plugin/tests -q -rs
5 passed, 3 skipped in 1.58s
SKIPPED test_real_load.py is a direct integration check: requires Hermes core and an installed plugin
SKIPPED test_routes.py ... requires live provider credentials and network access
SKIPPED test_provider.py ... requires Hermes core, provider credentials, and network access

The fix is better than the one we suggested. Rather than converting the scripts to pytest idioms, a conftest.py keeps them out of the import-time collector and test_scripts.py runs each offline script in its own subprocess — which also stops their module-level environment stubs leaking into each other — while the three that need Hermes core or live credentials become declared skips that say what they need. One of the new tests is test_script_inventory_is_explicit, which asserts the set of files on disk matches the two lists, so a script added later cannot quietly escape the classification. The README now documents both commands.

The benchmarks directory is unusually careful about its own risk: the README says the corpus is mined from your session history, that one command in the author’s own corpus held a working bot token, and that every artefact is git-ignored and redacted before writing.

Numbers, and whose they are

docs/METRICS.md reports the author’s baseline and then, at length, an independent third-party study that pinned a specific build, ran real guard preprocessing and real HTTPS calls, and reported timings, approval counts and four adapter defects it found — command-delimiter loss, silent policy truncation, a CLI-flag redaction overmatch and a conservative composition — all of which the author says are covered by regression tests in the next version. We have not reproduced either set of numbers, and this review makes no claim about them; what we can say is that publishing someone else’s findings about your own plugin, in your own metrics document, is rare.

MIT, 25 commits, and a 24-row holdout fixture of recorded answer sets (10 approve, 8 escalate, 6 deny) so the ladder can be exercised without calling anything.

Verdict

If you run Hermes Agent with smart approvals, this is a better reviewer than a prose-parsing one, and the reasons are structural rather than model-dependent: fixed rules over typed answers, injection checked first, DENY never softened, credentials redacted at the boundary, and every unclear case landing on the user.

If you don’t run Hermes, read plugin/jev_policy.py anyway — 67 lines is the whole policy, and it is the clearest example in this directory of a model informing a decision without being the decision.

For the same idea in coding agents, see Jevvy and pi-warden.

See how it compares with other tools in Best Jev tools, tested hands-on.

Review updated Sep 22, 2026. Numbers quoted from the project are its author's own; we don't publish our own measurements of Jev.