Using Jev as a LangSmith evaluator

Set it up

As of September 23, 2026, the setup steps come from LangChain's September 21 launch post ↗ and its screenshots. The LangSmith docs don't yet have a TypeSafe evaluator page; where they describe the same screens, their labels are used below.

TypeSafe not in the provider list? Organization admins can turn providers off under Settings → Model providers, and workspace admins control which providers each feature, Evaluators included, can use in Model configurations. LangChain's instructions name no LangSmith credential beyond the TypeSafe key. TypeSafe is bring-your-own-key in LangSmith: Gateway Credits and the free hosted SemIf model don't cover Jev.

State vs. questions

The launch post is explicit on this, and it's the main habit to unlearn from LLM judges. State is the evidence. Questions are the grading. A judge prompt that starts “You are an expert evaluator…” doesn't belong in state. TypeSafe's state guide ↗ draws the same line.

<customer_message>
{{input}}
</customer_message>

<agent_output>
{{output}}
</agent_output>

Map only what your questions need, not the whole trace. There are three reasons:

What you can map depends on what the evaluated run or thread records. If tool calls live only in child runs, they won't be in a root run's input or output. Check your own run in the evaluator's preview before relying on a field.

Questions and feedback keys

All questions in one evaluator go to Jev together, in one call per run or thread, and each comes back as a separate feedback key named after the question. LangChain notes that a second or third question adds only a little cost.

The recorded values come from LangChain's screenshot of a live run, which shows pii_leakage 0.03, intent "billing", and frustration 0.17. LangChain hasn't documented whether LangSmith stores a Score's expected level unchanged, or whether a Choice's probabilities and confidence survive in the metadata. Open metadata.typesafe on your first results before building on it. For picking a type, see Choice, Score, and Noul.

Example: a support agent

One evaluator on a customer-support agent's tracing project, with three questions for three different jobs. Each question asks about one thing, so each key can be read on its own.

State is the customer's message, the tools the agent called, and its final reply, mapped as in the template above. If the run you evaluate doesn't record its tool calls, log them there or drop the wrong_tool label. Account history, system prompts, and full retrieval dumps stay out unless a question needs them.

Downstream, these are ordinary LangSmith feedback keys. You can filter runs by them, chart them, raise a Feedback Score alert (average score over a time window), or fire a webhook automation. LangSmith recommends adding a feedback filter such as has(feedback_key, "resolved") to the webhook rule, so it fires only after the score exists.

Online, offline, and code

The TypeSafe provider path is documented for online evaluation, which scores production traces in a tracing project. That isn't the only way to put Jev in a LangSmith eval, but the other ways are different mechanisms.

For offline experiments today, write the judge yourself. This evaluator asks the same three questions as the example in one Jev call, through LangChain's langchain-typesafe package, and returns them as three feedback keys. Here you choose the state, can include reference outputs, and keep full probabilities if you want them.

from langsmith import Client
from langchain_typesafe import Choice, Noul, Score, TypeSafeClassifier

jev = TypeSafeClassifier()  # reads TYPESAFE_API_KEY

QUESTIONS = {
    "resolved": Noul(
        instructions="Does the final reply resolve what the customer asked for?"
    ),
    "failure_mode": Choice(
        instructions="Which best describes how this run went?",
        criteria={
            "none": "Handled correctly.",
            "unneeded_escalation": "Handed off to a human when it could finish.",
            "wrong_tool": "Called a tool that does not fit the request.",
            "unsupported_claim": "States account facts no tool result supports.",
        },
    ),
    "reply_quality": Score(
        instructions="How usable is the final reply for the customer?",
        criteria=[
            "Unusable: wrong, off-topic, or empty.",
            "Partial: on-topic but missing a needed step or detail.",
            "Complete: correct and actionable.",
        ],
    ),
}


def jev_judge(inputs: dict, outputs: dict) -> list[dict]:
    state = {
        "customer_message": inputs["message"],
        "tool_calls": outputs["tool_calls"],
        "final_reply": outputs["reply"],
    }
    r = jev.invoke({"state": state, "questions": QUESTIONS})  # one Jev call
    return [
        {"key": "resolved", "score": r.nouls["resolved"].noul},
        {"key": "failure_mode", "value": r.choices["failure_mode"].choice},
        {"key": "reply_quality", "score": r.scores["reply_quality"].score},
    ]


Client().evaluate(
    support_agent,         # your target function
    data="support-cases",  # your LangSmith dataset
    evaluators=[jev_judge],
)

Checked with langsmith 0.14.0 and langchain-typesafe 0.0.1a3 against a local mock of the TypeSafe API, with upload_results=False. It made one request carrying all three questions and produced the three feedback keys. The TypeSafe key here is used by your process, not by LangSmith's provider secret.

Jev judge vs. LLM judge

Same form, different machine. These are the operational differences, not a verdict on which is better.

The “suits” row is LangChain's own framing. Speed and cost multiples (“up to ~450x cheaper and ~200x faster”) are TypeSafe's figures for classification tasks, as quoted by LangChain. Measure yours on your own traces. For the underlying model difference, see Jev vs. LLMs.

Privacy and retention

In practice, map the fewest fields that answer your questions, and use the evaluator's filter to skip runs you know carry sensitive data. Whether any of this is acceptable for your data is a call for your own privacy and compliance review.

Thresholds

The LangSmith integration records the number; it applies no pass/fail cutoff, and this page doesn't suggest one. A Noul of 0.62 on resolved is a probability, not a failing grade. Where it should trip an alert depends on how often Jev agrees with your reviewers on your traces, and on which mistake costs you more.

To pick a cutoff, label a sample of real runs, for example in an annotation queue. Compare those labels with the Jev feedback on the same runs, then set the alert or filter threshold from that comparison. Averages over time (what a Feedback Score alert watches) catch drift. Per-run cutoffs catch incidents. They need different thresholds. The broader method is in calibrated decisions and the classification task.

The evidence so far

LangChain's Jev-as-a-Judge study ↗ (repo ↗) froze five weather-agent runs and had one human reviewer label them (four pass, one fail). Jev, GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 each judged every run 100 times. Jev matched the human pass/fail label on all 500 decisions, had the lowest score variance, and averaged 0.44 s and $0.00035 per call.

It's a promising, narrow result, and LangChain calls it “one test on one agent.” Four details matter if you read it as evidence for your own evaluator:

For how Jev FYI weighs vendor, partner, and independent results, see Benchmarks.

Other LangChain surfaces

“Jev in LangSmith” can mean three different things. Only the first is covered above.