Jev benchmarks: what the results actually show.

“Reproducible” means public code and enough method detail to rerun the study. It does not mean every published run exposes every raw prediction. Community leaderboard results remain separate because task authorship, composite scoring, and board revisions add additional judgment.

TypeSafe evaluates four code-defined workflows: security incidents, agent-trace observability, invoice processing, and customer service. Each model’s output is compared with consensus probability labels made from GPT-6 Astra and Claude Fable 5.1 at high thinking. The public overview reports an equal-weight mean across the four tasks.

BTZSC pilot: 300 zero-shot classifications

AbdelStark compared the same input text and ordered label descriptions with Jev and GLiNER2.5. The protocol was frozen before inference. Each dataset contributes 100 approximately class-balanced held-out examples.

Jev resolved to jev-1.13.0. Hosted Jev latency from France and local GLiNER CPU latency are not hardware-normalized. On Emotion, Jev assigned zero probability to the true label in 16% of examples; its Brier score was 0.846 versus 0.668 for GLiNER. The repository publishes aggregate JSON and artifact hashes, but not per-example probability files.

Classifier benchmark: 1,500 public-dataset examples

Dhruv Mehra’s harness sampled 500 examples each from SST-2, AG News, and Banking77. It compared Jev through OpenRouter with Laya, two prompted LLMs, zero-shot BART-MNLI, and DistilBERT fine-tuned on up to 10,000 training examples per dataset.

Model: typesafe/jev-1.13. This is a useful zero-shot deployment comparison, not a level training comparison: DistilBERT received supervised task data. The checked revision publishes code, configuration, and a summary, but not the run’s raw per-example result JSON. Latency includes each system’s different serving path.

Chess and NPC speech: task shape changes the answer

Eric Martz tested jev-latest, resolved asjev-1.13.0, on one task outside Jev’s intended shape and one close to bounded real-time routing. Code, labels, reports, and 1,337 raw request/response files are public.

JevBench’s current frozen set contains 534 English typed decisions: 72 easy, 96 standard, 146 answer-judging, and 220 hard items. Of the hard items, 109 are held out. The machine-readable artifact reports these Jev 1.13.0 component results:

These values describe one authored suite, not a universal rank. The leaderboard’s composite “JevBench Score” blends custom intelligence, calibration, speed, and cost formulas. Some competing endpoints also receive assumed latency adjustments. This page therefore preserves the measured Jev components and does not present the composite as “Jev accuracy.”

Latency was measured from one server in Germany at one time. The standard+judge run was serial. Public tasks may be known to models; held-out tasks were still sent to evaluated services. The suite is English-only and continues to add systems without changing the frozen task set.

Why two honest benchmark numbers can disagree

The Banking results are a useful example. The BTZSC pilot reports 87% Jev accuracy on 100 approximately class-balanced examples with 72 candidate labels. Dhruv Mehra’s run reports 76.4% on a different seeded sample of 500 examples with all 77 Banking77 labels. Different sample, label space, descriptions, and transport mean these are not replications of one another.

What Jev FYI has not measured yet

Jev FYI has not published its own Jev benchmark, calibration study, latency run, cost comparison, or reproduced failure ledger. Category E—Jev FYI first-party evidence—is empty. Every number on this page is attributed to its original publisher.

No result here establishes a global Jev accuracy rate, universal calibration, provider-independent latency, or expected production savings. A deployment decision still needs a labeled sample from the actual task, the exact Jev version and provider path, and thresholds selected on validation data before one held-out test.