Key Health & Re-authentication

Every provider credential in the vault carries a health state. SpiderGate uses it to decide whether a key is eligible to serve your next request, demotes keys that fail, ejects keys that are genuinely dead, and returns the recoverable ones to the pool on their own. This page is the exact state machine, the thresholds, and the endpoints involved.

For the dashboard walkthrough see The Key Vault.

Health states

A credential's health_status is one of:

• healthy — eligible for routing. The engine loads only healthy, active credentials into the live pool.

• degraded — transient trouble recorded, still eligible. No notification fires in this state.

• unhealthy — three or more consecutive failures. Routing skips the key, and the contributor notification episode begins.

• failed — terminal. Set by the refresh path when a credential can no longer be revived without human action.

Health is separate from is_active. is_active = false means the key is out of the pool entirely, whether an admin switched it off or SpiderGate deactivated it. Both flags must be favourable for a key to serve: the engine's fetch requires health_status = 'healthy' and is_active = true.

Thresholds

The numbers the state machine actually uses:

• Demote to unhealthy — 3 consecutive failures.

• Half-open recovery — a credential that has been non-healthy for 20 minutes, is still active, and has recorded no AuthenticationError or PermissionDeniedError in that window is reset to healthy and retried. Rate limits and timeouts recover this way.

• Terminal deactivation (api_key) — an already-unhealthy API-key credential that has recorded 3 or more authentication or permission errors within 24 hours is set is_active = false. It leaves the pool permanently, because recovery requires is_active = true. Only a re-authentication brings it back.

• Terminal deactivation (OAuth) — handled separately by the token-refresh path: 3 failed refreshes set is_active = false.

• OAuth recovery guard — an OAuth credential is only eligible for half-open recovery when its stored token_expires_at is more than 5 minutes in the future, so recovery never races the refresh loop.

• Sweep interval — every 10 minutes. There is no separate polling health check; health is driven by real request outcomes, and the sweep acts on what those outcomes recorded.

After every sweep the live router is reconciled against the current healthy set, so recovered keys re-enter and deactivated keys are ejected without a restart.

What a caller sees

Key health is invisible to a normal request. A demoted key is skipped and the next eligible key serves, which is the point of pooling. You only observe it in two cases:

• Fallback in a trace. When the first model in an alias chain has no healthy key, the request falls through and the trace records which model actually answered. See Traces.

• 503 no_openai_key or service_unavailable. Returned when no healthy credential exists for a required provider at all. This is the state key health exists to prevent, and it means every key for that provider is unhealthy or inactive. See Errors.

Key health never produces a distinct client-facing error code of its own.

Notifications

When a credential is unhealthy, the contributor who added it is emailed a re-authentication link:

• At most 3 emails per failure episode, at least 48 hours apart.

• Deferred during the recipient's quiet hours — the send waits rather than burning one of the three, so a reminder lands after the do-not-disturb window rather than overnight.

• Reset on recovery. When the key returns to healthy and records no fresh auth error, the episode counter clears, so a future failure notifies from scratch.

• Contributor only. Brand admins do not receive per-key mail; they receive one daily digest at 09:00 in their own timezone, listing every unhealthy key in the brand, split into newly failed and still unhealthy, each row naming the contributor to chase.

A credential with no resolvable contributor email still appears in the admin digest.

Re-authentication endpoints

These are public and token-authenticated. The token arrives in the emailed link, is valid for 120 hours, and is the only credential required — there is no login. The 120-hour lifetime deliberately outlasts the full three-reminder cycle.

Base path: https://spideriq.ai/api/v1/gate/reauth

A successful re-authentication updates the existing credential in place: same key id, same sharing and pool settings, same limits, same history. It clears the failure state and sets is_active = true, and the engine reloads so the key serves again without a restart.

GET /api/v1/gate/reauth/{token}

Details for the re-auth page: which provider and key label the link is for, why it failed, and which flow the contributor should be shown. No authentication beyond the token itself.

curl -s https://spideriq.ai/api/v1/gate/reauth/9f2c1d7e4b6a8035c1e2f4a6b8d0c2e4 \
  -H "Accept: application/json" | jq '.provider_name, .reason'

Response:

{
  "provider_name": "mistral",
  "key_label": "Mistral production",
  "auth_type": "api_key",
  "reason": "This key was auto-disabled after 3 consecutive failures.",
  "expires_at": "2026-08-15T09:14:22Z",
  "token_expired": false
}

Errors: 410 when the link has expired or was already consumed. 404 when the token matches no credential.

POST /api/v1/gate/reauth/{token}/api-key

Re-authenticate an API-key credential by supplying a fresh secret.

curl -s -X POST https://spideriq.ai/api/v1/gate/reauth/9f2c1d7e4b6a8035c1e2f4a6b8d0c2e4/api-key \
  -H "Content-Type: application/json" \
  -d '{"api_key": "kf7Qd2mXpR9vLzT4nB6wYc1sJ8hA3eG5"}'

Response:

{
  "success": true,
  "credential_id": 47,
  "provider_name": "mistral",
  "health_status": "healthy",
  "is_active": true
}

Errors: 400 when the provider canonically authenticates via OAuth, even if the stored auth_type says api_key — a subscription plan has no key to paste, so the contributor is routed to the re-login flow instead. 410 on an expired link.

POST /api/v1/gate/reauth/{token}/oauth/start

Begin the OAuth re-login for an OAuth or PKCE credential. Returns the authorization URL to send the contributor to.

curl -s -X POST https://spideriq.ai/api/v1/gate/reauth/9f2c1d7e4b6a8035c1e2f4a6b8d0c2e4/oauth/start \
  -H "Accept: application/json" | jq '.authorization_url'

Response:

{
  "authorization_url": "https://auth.openai.com/authorize?client_id=app_7Kd2&code_challenge=x9Fq2LmT",
  "state": "b41c8ae27fd3"
}

Errors: 400 when the credential is not an OAuth provider. 410 on an expired link.

POST /api/v1/gate/reauth/{token}/oauth/callback

Complete the re-login and store the refreshed token against the existing credential.

curl -s -X POST https://spideriq.ai/api/v1/gate/reauth/9f2c1d7e4b6a8035c1e2f4a6b8d0c2e4/oauth/callback \
  -H "Content-Type: application/json" \
  -d '{"code": "ac_3Nq8Rt5Wz2", "state": "b41c8ae27fd3"}'

Response:

{
  "success": true,
  "credential_id": 55,
  "provider_name": "codex",
  "token_expires_at": "2026-09-09T11:02:40Z",
  "is_active": true
}

Errors: 400 on a state mismatch or a rejected authorization code. 410 on an expired link.

POST /api/v1/gate/reauth/request-new-link

Mint a fresh link when the emailed one has expired. Accepts the expired token or the credential id. The new link is emailed only to the contributor already on record — never to an address supplied in the request, so there is no open-relay surface.

curl -s -X POST https://spideriq.ai/api/v1/gate/reauth/request-new-link \
  -H "Content-Type: application/json" \
  -d '{"token": "9f2c1d7e4b6a8035c1e2f4a6b8d0c2e4"}'

Response:

{
  "accepted": true,
  "message": "A new re-authentication link has been emailed to the contributor on record."
}

Errors: 404 when neither the token nor the credential id resolves. The response deliberately does not reveal the contributor's address.

Admin

• POST /api/v1/admin/gate/keys/{key_id}/reset-health — clear consecutive_failures and return a credential to the healthy pool. Requires admin authentication. Use it after fixing a key out of band; it does not replace the credential's secret, so a key that is genuinely revoked will simply fail again and be demoted.

The same action is available to agents as resetKeyHealth in the manage-vault skill.

Related

• The Key Vault — adding, sharing and re-authenticating keys in the dashboard.

• Errors — the full status-code reference, including 503.

• Traces — seeing which model actually served a request.