Rate Limits That Make Sense on a Data API

Every public data API eventually collides with its own success. The endpoint that served a hundred cautious queries a day now takes ten thousand, and the operator has to decide, often overnight, how much traffic the system can absorb before somebody else's request starts timing out. Rate limits are the instrument for that decision, and most of them are written badly. A limit that punishes the wrong caller, that resets on the wrong clock, or that hides its own rules becomes a tax on the people who take the service seriously.

The subject here is narrow and technical: the numbers and mechanics a data API uses to govern request volume. Leave aside authentication, billing tiers, and marketing pages. Focus on the counter itself, what it counts, when it resets, and what it tells a caller who has run into the ceiling.

A limit is a promise about capacity, and most APIs break it

Consider what a rate limit actually communicates. It says the operator has measured the system and believes it can sustain a certain load. When the limit is set at 100 requests per minute, the caller hears something honest: this service can handle 100 a minute from one account, sustained, without degradation. That number should come from load testing, from queue depth measurements, from the behavior of the slowest database query in the hot path.

In practice the number comes from a competitor's documentation, a guess made during a launch week, or a default left in a configuration file. A limit set at 60 requests per minute because the previous product used 60 is not a promise about anything. It is a number that happens to be written down.

The distinction matters because callers build retry logic, caching layers, and batch schedulers against the published figure. When the real capacity is much lower, the first traffic spike produces a cascade of 429 responses, and the callers who obeyed the documented rules are the ones who get punished first.

Fixed windows make liars of careful clients

The most common implementation counts requests inside a fixed one-minute window. At the top of each minute the counter resets to zero. This is simple to build and cheap to run, and it produces a predictable failure mode: a caller who has been pacing carefully for fifty seconds can spend the whole budget in the last ten and then spend it again in the first ten of the next window, doubling the instantaneous load.

From the server's point of view, the burst arrives all at once. From the caller's point of view, the published limit was never exceeded. Both are correct, and the disagreement is baked into the mechanism.

Sliding window counters, or token buckets with a refill rate, smooth this out. A token bucket that holds 100 tokens and refills at roughly 1.7 per second allows the same nominal budget while forbidding the boundary exploit. It also gives the caller a clearer mental model: tokens accumulate when idle, drain when busy, and the refill rate is the sustained throughput. That is a limit a scheduler can actually reason about.

Counting requests instead of work penalizes small queries

A request that returns a page of product listings and a request that triggers a full aggregation across a year of transactions are not the same load. Counting both as one unit against the same limit forces a caller to treat the cheap call and the expensive call as interchangeable. The result is predictable: careful callers underuse the expensive endpoint and overuse the cheap one, and the aggregate load on the system looks nothing like what the limit was designed to control.

Weighted limits fix this. Assign a cost to each endpoint, publish the cost, and count cost units rather than raw request counts. A simple lookup might cost one unit. A search across a large index might cost five. A bulk export might cost fifty. Callers then make rational decisions about which operations to batch and which to cache.

The same logic applies to payload size. A GET that returns 2 KB and a GET that returns 20 MB place very different demands on bandwidth, serialization, and downstream caching. Counting by bytes transferred, at least as a secondary limit, catches the caller whose request rate looks reasonable while the data volume is not. Without it, one client can saturate an egress link while reporting a request rate comfortably under the published ceiling.

Per-account limits do not protect a shared backend

Most rate limits are keyed to an individual API key or account. This is convenient for billing and reasonable for fairness between customers, and it is close to useless as protection for the backend. A single account with a generous limit can exhaust a shared resource, and every other account on the same cluster feels the consequences.

Effective designs layer limits at multiple scopes. Per-key limits enforce individual fairness. Per-tenant or per-organization limits prevent one customer from concentrating load. Global limits protect the infrastructure itself. The layers need to be published, because a caller who has been told the per-key limit and then hits an undocumented global ceiling has been misled about what the system will accept.

The tricky part is that global limits are shared, so a single misbehaving client can consume the budget for everyone. That argues for prioritizing within the global ceiling rather than refusing traffic indiscriminately. A caller who has been steady at 200 requests per minute for a month probably deserves more headroom than an anonymous key that appeared an hour ago and is already hammering the export endpoint.

A 429 without a Retry-After is a dead end

The response to a rate-limited call carries most of the information a caller needs to recover. Status 429 is the standard. The Retry-After header tells the client how long to wait before trying again, and the absence of this header is one of the most common causes of retry storms in otherwise well-behaved clients.

When the server does not say how long to wait, a client has to guess. The guesses cluster around a few defaults: one second, five seconds, exponential backoff starting at 100 milliseconds. None of them are informed by the actual state of the system. A thousand clients all guessing two seconds and retrying are indistinguishable from a thousand clients that never received a limit at all.

Headers that report remaining quota and reset time serve the same purpose proactively. A caller that knows the remaining budget can pace itself against the reset, which is far better than discovering the limit by hitting it. The published header set should include the limit, the remaining count, and the timestamp of the next refill. These are cheap to compute and they eliminate most of the guesswork.

Burst allowances should be separate from sustained rates

A caller who has been idle for an hour and then sends a batch of fifty requests is not the same as a caller sending fifty requests every minute. The idle caller is using headroom the system has, in a sense, already reserved. The steady caller is consuming capacity continuously. Treating both as the same load discards useful information.

Token bucket designs capture this distinction naturally. The bucket has a burst capacity (how many tokens can accumulate during idle periods) and a refill rate (the sustained throughput). Setting the burst capacity higher than the per-second refill rate allows short spikes from idle callers while keeping the long-run rate under control. Publishing both numbers gives callers a way to think about their own traffic shape.

Burst allowances also matter for interactive use. A dashboard that refreshes on user action sends traffic in bursts. A nightly batch job sends traffic in a steady stream. Both are legitimate patterns, and a limit that only acknowledges one of them will frustrate the other.

Limits need to be documented where the client will look

The published limit is only useful if it reaches the developer before the first 429. Documentation buried three clicks deep in a reference page under a heading like "Quotas and Limits" is routinely missed. The limit should appear in the error response, in the SDK defaults, and in the onboarding material. A test call that returns the current budget for the calling key is a small feature with an outsized effect on client behavior.

The documentation should also state the counter scope explicitly. Does the limit count per key, per IP, per user, or across all of them? A client that assumes per-key and discovers a per-IP ceiling has been given a wrong model of the system. Ambiguity here produces support tickets and, worse, calls that no one can reproduce because the limiting factor was never disclosed.

The cost of a limit set too low is paid in client rewrites

A limit that turns out to be too generous can be tightened once the operator understands actual usage. A limit that turns out to be too restrictive is much harder to fix, because every client has already built around the published figure. Raising it later helps new callers, but the existing integrations have already added caching layers, batch schedulers, and backoff logic sized to the old number, and many of them will not be revisited.

The asymmetry argues for starting slightly generous and monitoring the actual distribution of traffic. If a small number of callers sit right at the ceiling, the limit is probably too low. If the vast majority sit well below it and only outliers hit it, the limit is doing its job. That distribution is the only honest signal for whether the published number reflects anything real.

A rate limit that respects the caller is a small piece of engineering with a long tail. It shows up in how fast integrations ship, how often the support queue fills with retry complaints, and whether the developers who depend on the API describe it as fair or as a constant fight. The mechanics are not glamorous, but the difference between a limit that holds and one that gets circumvented is mostly a matter of whether the operator treated the number as a promise or as a placeholder.