Rate Limiting for AI Agents: Why Token Buckets Beat Simple Throttling
When you're running 200 agents through a single gateway, naive rate limiting — "max 100 requests per minute" — creates more problems than it solves. Agents burst, queue, timeout, and retry, creating cascading failures that amplify the original constraint.
The Problem with Simple Rate Limits
Fixed-window rate limiting treats all requests equally:
A health check ping counts the same as a 4,000-token completion
A burst of 50 requests in 1 second is identical to 50 requests spread over 60 seconds
Priority agents wait behind non-critical background tasks
Token Bucket Algorithm
SpiderGate uses an adaptive token bucket approach:
How It Works
Each agent gets a "bucket" that fills with tokens at a configurable rate. Each request consumes tokens proportional to its cost (estimated input + output tokens). When the bucket is empty, requests queue until tokens replenish.
Why It's Better
Cost-proportional — expensive requests consume more capacity
Burst-friendly — agents can burst up to their bucket capacity
Fair scheduling — high-priority agents get larger buckets
Adaptive — bucket refill rates adjust based on provider capacity
Priority Tiers
SpiderGate supports 4 priority tiers:
| Tier | Use Case | Bucket Size | Refill Rate | |------|----------|-------------|-------------| | Critical | Customer-facing agents | 10,000 tokens | 2,000/s | | High | Revenue-impacting workflows | 5,000 tokens | 1,000/s | | Normal | Standard automation | 2,000 tokens | 500/s | | Low | Background/batch processing | 500 tokens | 100/s |
Backpressure Signals
When an agent's bucket runs low, SpiderGate can:
Queue — hold the request and process when capacity is available
Downgrade — route to a cheaper model automatically
Reject — return a 429 with retry-after header
Alert — notify the team that an agent is hitting limits
Results
Teams migrating from simple rate limits to token buckets see:
3x improvement in throughput under load
60% reduction in timeout errors
Fair resource distribution across agent tiers
Token bucket rate limiting is configurable per-agent, per-team, or globally.