July 11, 2026

Multi-Provider LLM Failover: A Circuit Breaker Across Five Inference Providers

Inference providers go down, rate-limit, and degrade. I run a failover chain across SambaNova, Groq, Gemini, OpenAI, and Cerebras — here is why a retry loop is not enough and what a circuit breaker actually buys you.

aillmreliabilitycircuit-breakervercel-ai-sdk

If your product depends on a single LLM provider, your uptime is that provider's uptime. For most demos that is fine. For a tax compliance platform that users rely on at filing deadlines, it is not.

REPSShield runs AI-assisted real-estate tax compliance for US clients — agentic time-tracking, RAG advisory over IRS documentation, and an audit simulation. All three modes are inference-dependent. When the provider is down, the product is down.

So inference runs through a failover chain across five providers: SambaNova → Groq → Gemini → OpenAI → Cerebras. This post is about why that chain needs a circuit breaker rather than a retry loop, which is the part most people get wrong.

Why a retry loop is not enough

The instinct is to wrap the call in a retry: if provider A fails, try provider B.

This works for a transient failure — one bad request, one dropped connection. It fails badly for the failure mode that actually matters, which is a provider being down or rate-limiting you for the next twenty minutes.

In that case, a naive chain means every single request pays the full cost of discovering that A is broken before falling through to B:

You wait for A's timeout. Every time. If A is hanging rather than erroring, that is 30 seconds of latency added to every request for the duration of the outage.
You keep hammering a provider that is rate-limiting you, which in some cases extends the rate limit.
Your p99 latency collapses even though a perfectly healthy provider B is sitting right there.

The retry loop has no memory. It rediscovers the same outage on every request.

What the circuit breaker adds

A circuit breaker gives the chain memory. Each provider has a state:

| State | Meaning | Behavior | |---|---|---| | Closed | Healthy. | Requests flow normally. | | Open | Failing. | Requests skip this provider entirely — no call, no timeout, no waiting. | | Half-open | Probation. | After a cooldown, let one request through to test recovery. |

The transition that matters is closed → open: after N consecutive failures, the provider is marked down and taken out of the chain. Subsequent requests do not attempt it at all. They go straight to the next healthy provider, with none of the timeout cost.

After a cooldown, the breaker goes half-open and admits a single probe request. Succeed, and it closes and rejoins the chain. Fail, and it opens again and the cooldown resets — typically with backoff, so a provider in a long outage is probed less and less often.

async function withFailover(providers, request) {
  for (const provider of providers) {
    if (breaker.isOpen(provider.name)) continue  // <- the whole point

    try {
      const result = await provider.call(request)
      breaker.recordSuccess(provider.name)
      return result
    } catch (err) {
      breaker.recordFailure(provider.name)
      if (!isRetryable(err)) throw err   // don't fail over on a bad prompt
      continue
    }
  }

  throw new AllProvidersUnavailableError()
}

The continue on an open breaker is doing the real work. It is the difference between an outage costing you 30 seconds per request and costing you nothing.

Not every error should trigger failover

This is the subtlety that bites people. A failing request is not the same as a failing provider.

If you send a malformed prompt, exceed a context window, or trip a content filter, every provider in your chain will reject it — identically. Failing over on those errors means you burn all five providers on a request that was never going to succeed, and worse, you record five spurious failures and may trip five breakers open on a bug in your own code.

So errors have to be classified:

Fail over: timeouts, 5xx, 429 rate limits, connection resets. The provider is the problem.
Do not fail over: 400s, context-length errors, content-policy refusals. You are the problem, and the next provider will tell you the same thing.

Getting this classification wrong is how a single bad input takes down your entire inference layer.

Ordering the chain

The order is a cost/latency/quality decision, not an arbitrary one. Fast, cheap inference providers sit at the front; the expensive, highly-available incumbents act as the backstop at the end.

The thing nobody mentions: outputs differ

A failover chain quietly assumes the providers are interchangeable. They are not.

The same prompt against Groq and against Gemini returns differently-shaped output — different formatting habits, different adherence to a JSON schema, different tool-calling reliability. If your application parses the response, a failover can hand you a response your parser has never seen.

This is why structured output and schema validation matter more in a multi-provider setup than a single-provider one: the schema is what makes providers substitutable. Without it, failover does not degrade gracefully — it degrades silently, which is worse.

The takeaway

Multi-provider failover is not "try the next one." It needs three things a retry loop does not have: memory (a breaker, so an outage costs you nothing after the first discovery), error classification (so your own bad request does not trip all five providers), and a schema (so the providers are actually substitutable).

Get those right and provider outages become invisible to your users. Skip them and you have built a system that fails five times slower than a single provider would have.

I'm Ehnand Azucena — full stack and lead developer. I build the AI backend for REPSShield, an AI-powered real estate tax compliance platform: React 18, Node/Express, PostgreSQL 16, Vercel AI SDK v4, with agentic tool-calling, RAG over 282 IRS documents, and a 14-check audit simulation. I take remote contract and lead engagements — get in touch.