A provider key does not tell you it has died. It just starts returning 401s.
If that key is sitting in a shared pool serving a fleet of agents, the failure is almost invisible. Requests don't stop — they fall through to whatever is next in the chain. The work still happens. It just quietly happens on a weaker model, at a different price, with slightly worse output, and nobody gets paged because from the outside nothing is down.
That is the failure I care about, and it is the one most key pools handle worst. A dashboard that shows you a red badge is only useful if you happen to be looking at it. At 3am, nobody is looking at it.
So I stopped treating key health as something to display, and started treating it as something the gateway acts on by itself.
A failing key removes itself
The first job is to stop a broken key taking a share of your traffic.
Three consecutive failures mark a credential unhealthy, and routing simply skips it. Nothing about the request path changes — the rest of the pool absorbs the work, and your agents carry on. The key has stepped out of the lane rather than staying in it and dropping every request that lands on it.
This is the part that makes pooling worth doing. One dead key in a pool of fifty should be a non-event, and it only is if the pool notices faster than a person would.
Not every failure is a death
The mistake I made in the first version was treating every failure the same way, and it produced a horrible oscillation: a key would trip, get pulled out, get put back twenty minutes later, fail again, and cycle roughly hourly. Each cycle poisoned a slice of that provider's traffic.
The fix was to separate two things that look identical in the logs and are completely different in nature.
A transient failure — a rate limit, a timeout, a provider having a bad minute — heals on its own. So a credential that has been unhealthy for twenty minutes with no authentication errors goes straight back into the pool automatically. No human involved, because no human is needed.
A revoked or wrong key never heals. If a credential fails authentication three times inside twenty-four hours, it is deactivated outright and leaves the pool for good. Recovery requires an active credential, so it cannot claw its way back in and restart the cycle. Only a person with a working secret gets it back.
Telling those two apart is most of the value. Retire the transient failures and you throw away good keys; keep retrying the dead ones and you get the oscillation.
Tell the one person who can fix it
Then there's the part I got wrong in a more embarrassing way.
The first notification design emailed brand admins about every failing key. On a pool with real churn that turned into fifty emails a day for one person, several of them at two in the morning. A notification that arrives fifty times a day is not a notification; it's a filter rule waiting to be written.
So it splits. The contributor — the person who actually added the key and can actually replace it — gets a re-authentication link. At most three times, 48 hours apart, and never inside their quiet hours; the send waits for their morning rather than burning one of the three attempts at 02:46. Admins get one digest a day at 9am in their own timezone, listing every unhealthy key in the brand and naming who to chase for each.
The rule I'd extract from this: route an alert to the person who can act on it, and give everyone else a summary. Broadcasting an actionable alert to people who cannot act on it trains all of them to ignore it.
Fix it in place, not by rebuilding it
The last piece is what happens when the contributor clicks the link.
The obvious implementation is "delete the dead key, add a new one." It is also wrong, because a key in a pool is not just a secret — it carries sharing settings, limits, a usage policy, quotas and history. Rebuilding it means reconstructing all of that from memory, and getting one setting wrong silently changes how the pool behaves.
So re-authentication updates the existing credential in place. Same key id, same sharing, same limits, same history — only the secret changes. Paste a fresh key, or click through an OAuth re-login for a subscription provider, and the engine reloads without a restart.
The link is good for 120 hours. That number isn't arbitrary: the reminder cycle runs up to three emails 48 hours apart, so the link has to outlive the whole cycle or the last reminder would arrive carrying a dead link. And if it does expire, the expired-link page issues a fresh one to the contributor on record — no admin required, and no way to redirect that link to an address supplied in the request.
What it looks like in practice
The keys this catches are not exotic. A trial expires. A card declines. Someone rotates a secret on the provider side and forgets which pool it was in. A subscription provider revokes a session server-side without telling anyone — which is how I lost a Codex credential for seven days without noticing, and why any of this exists.
None of those are outages, and that is the whole point. Each one used to sit in the rotation failing a slice of requests, and the failures got blamed on the models.
The thing I'd want from a gateway is not that it never has a dead key. Keys expire, cards decline, tokens get revoked — that is normal. What I want is for the dead key to be out of the pool before it costs me anything, and for the one person who can fix it to already have the link in their inbox.
If you want the exact thresholds and endpoints, they're in the key health reference. If you have a key down right now, bring a failed key back is the shorter read.