Tasks / Model Routing

Model routing with Jev.

Using TypeSafe Jev as a model router allows engineering teams to evaluate incoming user requests and dispatch them among deterministic code paths, specialized generative LLMs, or human review queues.

By evaluating intents using Jev's Choice primitive and gating actions on calibrated confidence scores, developers can enforce deterministic control flow before incurring the cost or latency of large generative models.

The Routing Architecture Pattern

In traditional generative pipelines, developers often route requests by asking an LLM to output a JSON classification object. However, this approach can suffer from JSON parsing failures, high time-to-first-token latency, and uncalibrated confidence scores.

A System One routing architecture separates decision logic from generative text generation:

01State aggregation: Package the incoming user prompt, account metadata, permissions, and session history into a structured state string.

02Parallel classification: Send the state to Jev with a Choice question (identifying the target execution engine) and optional Noul questions (evaluating complexity, risk, or urgency).

03Confidence gating: Inspect the returned confidence float. If the model's uncertainty exceeds your application's tolerance, branch to a fallback queue or human review.

04Downstream dispatch: Route high-confidence requests directly to the selected destination (e.g. database lookups, cached answers, or specialized downstream LLMs).

Decision Topology

Destination Route	Dispatch Criteria	Downstream Execution
deterministic_lookup	Factual queries, status checks, FAQs, structured lookups.	Local database / Redis cache / Search index ($0 LLM cost).
generative_reasoning	Creative writing, multi-step problem solving, synthesis.	Invoke specialized generative model or reasoning LLM.
human_review	Confidence below threshold, policy ambiguity, or edge cases.	Triage queue for human operations or customer support.

Complete TypeScript Router Implementation

The following runnable implementation demonstrates end-to-end routing with confidence-gated branching and fallback handling:

TypeScript (Confidence-Gated Router Implementation)

import { TypeSafeClient } from "@typesafe-ai/sdk";

const client = new TypeSafeClient({
  apiKey: process.env.TYPESAFE_API_KEY!,
});

// Example downstream handler interfaces
interface RouteResult {
  route: string;
  source: "deterministic_cache" | "downstream_llm" | "human_review";
  payload: unknown;
}

export async function routeUserRequest(userPrompt: string, userTier: string): Promise<RouteResult> {
  // 1. Construct application state block
  const state = `User prompt: "${userPrompt}"\nUser account tier: ${userTier}\nSystem context: SaaS workflow management engine.`;

  // 2. Query Jev using a Choice primitive for intent & a Noul primitive for complexity
  const response = await client.systemOne({
    model: "jev-latest",
    state,
    questions: {
      target_engine: {
        type: "choice",
        instructions: "Select the most appropriate execution engine for this request",
        criteria: {
          deterministic_lookup: "Simple FAQ, status check, or database query",
          generative_reasoning: "Complex multi-step writing, code gen, or synthesis",
          unsupported_escalate: "Ambiguous, abusive, or unsupported requests"
        }
      },
      requires_deep_reasoning: {
        type: "noul",
        instructions: "This request requires complex multi-step reasoning rather than simple factual lookup."
      }
    }
  });

  const targetDecision = response.answers.target_engine;
  const isComplex = response.answers.requires_deep_reasoning.noul;

  // NOTE: This threshold (0.85) is an illustrative control-flow example.
  // In production, thresholds must be empirically determined against your labeled domain workload.
  const CONFIDENCE_CUTOFF = 0.85;

  // 3. Confidence-aware fallback: Route to human review if uncertain
  if (targetDecision.confidence < CONFIDENCE_CUTOFF) {
    console.warn(`Low router confidence (${targetDecision.confidence}). Escalate to human review.`);
    return {
      route: targetDecision.choice,
      source: "human_review",
      payload: { reason: "low_confidence", score: targetDecision.confidence }
    };
  }

  // 4. Dispatch based on calibrated decision
  switch (targetDecision.choice) {
    case "deterministic_lookup":
      return {
        route: "deterministic_lookup",
        source: "deterministic_cache",
        payload: await executeDatabaseLookup(userPrompt)
      };

    case "generative_reasoning":
      return {
        route: "generative_reasoning",
        source: "downstream_llm",
        payload: await invokeDownstreamLlm(userPrompt, isComplex > 0.70)
      };

    case "unsupported_escalate":
    default:
      return {
        route: "unsupported_escalate",
        source: "human_review",
        payload: { reason: "unsupported_intent" }
      };
  }
}

// Mock downstream handlers
async function executeDatabaseLookup(prompt: string) {
  return { status: "cached_response", query: prompt };
}

async function invokeDownstreamLlm(prompt: string, useHeavyModel: boolean) {
  const selectedModel = useHeavyModel ? "heavy-reasoning-model" : "fast-completion-model";
  return { status: "generated", modelUsed: selectedModel, prompt };
}

Calibrating Confidence Thresholds

Workload-specific validation: There is no universal confidence threshold that applies to all production systems. The threshold chosen in your codebase represents a deliberate trade-off between automation rate and error tolerance.

•Asymmetric risk profiles: In low-stakes tasks (such as selecting an email template), a lower confidence threshold may be acceptable to maximize automated throughput. In high-stakes tasks (such as authorizing account deletions or financial refunds), the threshold should be set higher, routing marginal cases to human review.

•Measuring empirical accuracy: Because Jev is trained via RLCD (RLCD guide), its confidence scores correlate with empirical error rates across groups. Teams should validate the distribution of confidence scores against their own labeled historical test sets before setting production cutoffs.

Handling Router Fallbacks & Failures

Robust production routing systems should account for two failure modes:

01Low-confidence predictions: When Jev indicates ambiguity below your workload's calibrated cutoff threshold, avoid making a blind guess. Escalate the request to a default safe handler, a general-purpose fallback model, or a manual review inbox.

02Upstream rate limits or timeouts: In the event of network timeouts or HTTP 429 rate limit errors (see documented limits on the API Access Guide), wrap the routing call in an exponential backoff loop or default to a conservative fallback route.

← Explore Decision Primitives (Choice, Score, Noul)Deploy Router on Vercel AI Gateway →Compare Jev to Generative LLMs →