Jev Confidence Thresholds: How to Choose Them

Every Jev Choice and Score answer comes with a confidence value from 0 to 1. The point of that number is to let your code act on the answers it can trust and escalate the rest. But a confidence score only becomes useful once you know what accuracy each level of it corresponds to on your data. That mapping is what a threshold encodes.

Why you have to measure it yourself

TypeSafe trains Jev so that its probabilities are calibrated: across many predictions, higher confidence should mean higher accuracy. Calibration is a statistical property, though, and it depends on the data. A question about banking intents and a question about academic abstracts can behave very differently at the same confidence level. The community project Janus found exactly that when it compared datasets.

TypeSafe’s own confidence docs say the same thing: the correct threshold “depends on your domain and the performance of the model for your use case.” So the workflow below is not optional polish. It’s the step that turns Jev from a demo into something you can automate on.

Step 1: Build a small labeled set

Collect real inputs from the workflow you want to automate, not synthetic ones, and label the correct answer for each question.

• Size: a few hundred examples per question is enough to start.
• Coverage: include the awkward cases (ambiguous tickets, mixed sentiment, edge-of-policy requests). Those are where thresholds matter.
• Freeze it: keep this set fixed so you can re-run it when anything changes.

Step 2: Run it and record everything

Pin the model version so your results mean something later, and store the full answer, not just the winning option.

from typesafe_sdk import Choice, TypeSafeClient

client = TypeSafeClient(model="jev-1.13.0")  # pin the version you're tuning

question = Choice(
    instructions="Which team should handle this ticket?",
    criteria={
        "billing": "Payments, invoices, refunds, subscriptions",
        "technical": "Bugs, errors, integrations",
        "account": "Login, access, profile changes",
    },
)

results = []
for example in labeled_set:  # [{"text": ..., "label": ...}, ...]
    answer = client.system_one(state=example["text"], questions={"team": question}).answers["team"]
    results.append({
        "confidence": answer.confidence,
        "predicted": answer.choice,
        "correct": answer.choice == example["label"],
    })

Step 3: Measure accuracy by confidence

Now answer one question: if you only act on answers above threshold t, how accurate are those answers, and what share of traffic do they cover?

def sweep(results, targets=(0.90, 0.95, 0.98)):
    ranked = sorted(results, key=lambda r: r["confidence"], reverse=True)
    for target in targets:
        best = None
        correct = 0
        for i, r in enumerate(ranked, start=1):
            correct += r["correct"]
            if correct / i >= target:
                best = (r["confidence"], i / len(ranked), correct / i)
        if best:
            t, coverage, acc = best
            print(f"target {target:.0%}: threshold {t:.3f}, handles {coverage:.0%} of traffic at {acc:.1%} accuracy")
        else:
            print(f"target {target:.0%}: not reachable on this question")

sweep(results)

This gives you the core trade-off in one table. A strict accuracy target usually means a high threshold and low coverage: more traffic goes to a person or a stronger model. If a target is unreachable, the fix is usually the question, not the threshold. Reword the instructions, sharpen the option descriptions, or split the question in two.

Step 4: Set thresholds by risk, not one number for everything

The same question can drive actions with very different costs of being wrong. TypeSafe’s docs recommend gating each action at its own level:

Action	Cost of a wrong call	Threshold
Show a suggested category	Low, and easy to undo	Low
Route a ticket to a queue	Medium	Medium
Issue a refund, delete data, move money	High, hard to undo	High, plus confirmation

A useful default shape is three bands: act automatically above the high cutoff, ask for confirmation or review in the middle, and hand off to a person or an LLM below the low cutoff.

Step 5: Handle Noul answers differently

Noul answers don’t have a confidence field. The noul value is itself the probability that the statement is true, so use two cutoffs:

p = response.answers["is_refund_request"].noul

if p >= yes_cutoff:
    handle_refund()
elif p <= no_cutoff:
    continue_normal_flow()
else:
    escalate()  # the model isn't sure either way

Tune yes_cutoff and no_cutoff separately with the same sweep. Don’t reuse a threshold tuned on a Noul for a Choice version of the same question. TypeSafe notes the two question types aren’t directly comparable.

Step 6: Price the escalations

Whatever falls below your threshold still costs money, either a person’s time or an LLM call. Multiply the escalation rate from Step 3 by that cost to see the real price of your setup. Loosening a single over-strict question is often the biggest cost lever. Our cost calculator helps with the LLM side.

Step 7: Keep it honest over time

• Pin versions. jev-latest moves when TypeSafe ships a new release. Pin jev-1.13.0 (or whatever you tuned on) and upgrade deliberately.
• Re-run your set on every upgrade and compare before switching.
• Watch the confidence distribution in production. If the share of low-confidence answers jumps, your inputs have probably changed.

jevcal automates Steps 3 to 7: it picks thresholds for an accuracy target, estimates how much traffic still needs an LLM, and can fail CI when a model update shifts the numbers.

A note on sharing your results

Keep your measurements internal. TypeSafe’s Master Customer Agreement says customers may not “publish benchmarks or performance information about the Services.” Your thresholds and error rates are for tuning your own system, not for a blog post.

Frequently asked questions

What is a good default confidence threshold for Jev?

There isn't a universal one. TypeSafe's docs say the right values depend on your domain and on how the model performs on your use case, and recommend starting conservatively and adjusting on your own data. Use a placeholder like 0.8 only until you have measured.

Does a Noul answer have a confidence score?

No. Choice and Score answers include a confidence value; a Noul returns only the probability that the statement is true. Threshold the noul value directly, ideally with separate cutoffs for yes and no and an uncertain band in between.

How many labeled examples do I need?

A few hundred real examples is a practical starting point for one question. You need enough examples in the high-confidence band to estimate its accuracy, so add more if most answers land there.

Can I publish the accuracy numbers I measure?

Check TypeSafe's Master Customer Agreement first. As of September 2026 it says customers may not publish benchmarks or performance information about the service, so keep evaluation results internal unless TypeSafe agrees otherwise.