How a multi-LLM
router cut our
AI agent bill by 80%.

The problem with single-model AI agents in production

If you ship an AI sales agent in 2026 and pin it to a single model — say, GPT-4o for everything — three things go wrong, in order, within the first month:

1. Cost explodes. GPT-4o is great at long reasoning. It is also good at extracting a phone number from a sentence, which is a job Gemini Flash would do for 1/40th the price. The bill scales linearly with volume, and the waste is enormous.
2. Latency spikes. When your model provider has a bad day — a regional degradation, a rate limit, a Friday afternoon traffic surge — your agent's first-reply latency spikes. For a sales agent, latency is a conversion problem, not just a UX problem.
3. Vendor lock-in compounds. Once your prompts, your eval suite, and your fine-tunes are baked into a single provider's API quirks, switching costs grow non-linearly. You are not choosing a model; you are renting a stack.

At RunThrou we hit all three in the first quarter of shipping AI sales agents. The fix was not a better model. The fix was a router — a small, fast piece of middleware that sits between the agent and the model providers and decides, for every request, which model to use, in what order of preference, with what fallback rules. We called it the multi-LLM router. Within four weeks of going live, the bill dropped 80%, median request latency dropped 45%, and we have not had a customer-facing model-downtime event since. The same architecture ships in the Studio Alpha infrastructure we built for high-availability AI routing, and it is the default layer behind every RunThrou AI agent build.

What the router actually decides, per request

A production AI sales agent is not one model call. It is a sequence of calls, each with a different shape. The router treats each call as a separate routing decision. Here is the breakdown for a typical WhatsApp lead-to-meeting pipeline:

Before the router, every one of those calls went to the same model — usually GPT-4o, because it was the safe default. The bill was 4-6x what it needed to be. After the router, the average cost per qualified conversation dropped from roughly $0.42 to $0.084, and the latency improved because cheap, fast models handle most of the volume and only the hard turns escalate.

The architecture, end-to-end

The router is a stateless service in front of the model APIs. It does not own conversation state — that lives in the agent's own memory layer. The router owns three things: policy, health, and metering. Here is the runtime flow for a single request:

1. Receive the call. The agent sends a request envelope: task type, prompt, expected output schema, latency budget, and a list of model preferences.
2. Classify the task. The router checks the task type against its policy file. "Intent classification" → Gemini Flash. "Structured extraction" → GPT-4o-mini. "Multi-turn dialogue" → Claude Sonnet. "Long-context summary" → DeepSeek. The policy is a 200-line YAML that the team tunes weekly.
3. Health-check the providers. A 100ms parallel probe checks the target provider's status. If healthy, dispatch. If degraded, mark and reroute to the next preference in the chain.
4. Dispatch with timeout. The router fires the request with a hard p99 timeout. If the provider misses, the request fails over to the next preference in <100ms.
5. Meter and log. Every request emits a metric: provider, model, task, tokens in/out, latency, cost, fallback count. The metric lands in the live dashboard; the team uses it to tune the policy file.

Here is a sketch of the policy file, simplified:

That is the entire policy. The interesting part is not the policy — it is the telemetry layer that tells you when to update it.

Why this is reliability engineering, not just cost engineering

The 80% bill reduction is the part everyone quotes. The more important number is the 99.99% uptime with sub-100ms failover. Here is what that buys you in a production AI sales agent:

• Zero customer-facing downtime during provider incidents. When Anthropic had a 12-minute degradation in March 2026, our agents never missed a beat — the router shifted dialogue generation to GPT-4o and the conversation quality delta was below the human-perception threshold for our clients' customers.
• No rate-limit outages. OpenAI's per-org rate limits are a real ceiling. The router fans across orgs, so a viral spike on one channel does not cripple the others.
• Defensible SLAs. We can offer clients a 99.99% uptime SLA on the AI agent layer, because we are not betting the SLA on a single provider's status page.

This is the same architecture we shipped for Studio Alpha, the multi-provider AI routing middleware RunThrou built for high-availability production pipelines. Studio Alpha holds 99.99% uptime in production with zero customer-facing model-downtime events since launch.

The cost math, in real numbers

Let us run the numbers for a representative RunThrou client — a real estate brokerage in Mumbai doing 8,000 WhatsApp conversations a month with an AI sales agent.

Before the router (single-model, GPT-4o for everything):

• Average prompt + completion per turn: ~1,200 tokens
• Average turns per conversation: 6
• Total tokens: 8,000 × 6 × 1,200 = 57.6M tokens / month
• GPT-4o blended rate: ~$5 / 1M tokens
• Monthly bill: ~$288

After the router (task-routed, mixed models):

• Intent classification (2 turns × 200 tokens, Gemini Flash): 3.2M tokens × $0.075 = $0.24
• Structured extraction (1 turn × 400 tokens, GPT-4o-mini): 3.2M × $0.15 = $0.48
• Dialogue generation (2 turns × 800 tokens, Claude Sonnet): 12.8M × $3.00 = $38.40
• FAQ lookup (1 turn × 300 tokens, Gemini Flash): 2.4M × $0.075 = $0.18
• Long-context summary (0.2 turns × 2,000 tokens, DeepSeek): 3.2M × $0.27 = $0.86
• Handoff message (1 turn × 200 tokens, Claude Haiku): 1.6M × $0.25 = $0.40
• Monthly bill: ~$40.56

That is an 86% reduction on the same conversation volume, same quality bar, and a measurably faster agent. The numbers line up with what we see across the rest of our client base — the headline 80% is conservative.

What the router does not solve (and what to do instead)

A multi-LLM router is not a silver bullet. Here are the limits we have hit, and the workarounds:

1. Eval drift. When you switch providers, the model behaviour drifts. Your "good response" eval set on Claude will not match GPT-4o 1:1. You need a per-task eval suite per provider, and you need to re-run it every time a model version changes. The router ships with a benchmark harness that runs nightly.
2. Latency variance. The router's p99 latency is the worst p99 across the providers in the chain. If one provider is consistently slow on a specific task, drop it from that task's preference list. The telemetry will tell you within a week.
3. Context handoff. Multi-turn conversations need a stable context. The router does not own the context — the agent does — but you must design the agent so that task boundaries are clean. Do not let a "dialogue" turn silently include a "long-context summary" because the prompt crept.
4. Compliance. Some clients have data-residency rules. The policy file lets you pin a task to a specific region (e.g., Claude on AWS, GPT-4o on Azure) without code changes. This is non-optional for our EU and UAE clients.

A 14-day plan to ship your own multi-LLM router

You do not need a six-month build to get the win. Here is the minimum viable router RunThrou ships in the first two weeks of a new client engagement, before any custom agent work:

1. Day 1-2: Pull the last 30 days of model usage. Classify every call by task. Build a CSV of (task, model, tokens, cost, latency). You cannot route what you have not measured.
2. Day 3-4: For each task type, pick the cheapest viable model. Run an eval set (50-100 prompts, scored by an LLM-as-judge) on each candidate. Keep the model that wins on quality; the router will handle cost.
3. Day 5-7: Build the router as a thin proxy in front of the model APIs. The policy file is the source of truth. Ship a single-task route first (e.g., FAQ lookup → Gemini Flash). Measure cost and latency for a week.
4. Day 8-10: Add failover rules and a health-check probe. Sub-100ms probe to the provider, sub-100ms failover on timeout. Verify with a chaos test: kill the primary provider, watch the agent keep working.
5. Day 11-14: Add the second and third task routes. Wire the dashboard. Set up an alert when a provider's p99 latency regresses 50% week-over-week. Hand the keys to the team.

That is the RunThrou default router. If you want it shipped as part of a full AI agent build — for a service business, a sales team, or a custom workflow — we are happy to talk. The starting point for a RunThrou build is ₹1,50,000, and the typical lead-to-revenue system ships in 21-30 days.

Field notes & further reading

This is the technical companion to the operating essay Speed to lead: why the first 5 minutes decide your pipeline. If you are building a router yourself, the most useful reference is the Studio Alpha case study, which walks through the same architecture in production. For a broader view of the AI agency model and pricing, the RunThrou agency page and the FAQ are the place to start.

Frequently asked questions