grilling

"100 per minute" is underspecified: - **Fixed window** allows 2x bursts at window boundaries (100 at 12:00:59, 100 at 12:01:00). - **Sliding window counter** or **token bucket** is smoother. Token bucket also lets you separate sustained rate from burst allowance. - Do you want a short-term cap too (e.g., 20/sec)? 100 requests in one second might hurt your backend even though it's "within limits." ## 3. If you add Redis, decide these up front - **Failure mode:** If Redis is down, do you fail open (no limiting) or closed (outage)? Usually fail open, with a local per-server fallback limit and an alert. - **Atomicity:** Use a Lua script or an atomic command. A naive `INCR` then `EXPIRE` can leave keys with no TTL if the process dies between them. - **Clock source:** Don't rely on app-server clocks for window boundaries. Use Redis time or TTL-relative logic. - **Latency budget:** This adds a network hop to every request. Set a tight timeout (e.g., 5–10ms) and fail open past it. ## 4. Things the draft doesn't cover - **Invalid or missing keys:** If you only limit by valid key, attackers can hammer you with random keys. Add a coarse per-IP limit *before* key lookup, and don't create counters for unrecognized keys (memory growth). - **Per-key vs per-account:** If enterprise customers or free users can make multiple keys, per-key limits are trivially bypassed. Decide what the billing/abuse unit actually is. - **Tier lookup:** Don't hardcode "enterprise = 1,000." Store limits per key or plan, cache them, and allow per-customer overrides. Someone will ask for 2,500 within a month. - **Request cost:** A cheap GET and an expensive search or export count the same. Consider weighted costs or separate limits for expensive endpoints, plus a concurrency limit. - **Do 429s count?** If rejected requests consume quota, a client that retries aggressively never recovers. Usually they shouldn't count, but then you need to be sure the rejection path is cheap. - **Volumetric abuse:** Per-key limiting isn't DDoS protection. That belongs at the CDN/WAF/edge. ## 5. Client-facing behavior - `Retry-After` must be accurate. With local counters it's wrong by construction. With token bucket, compute time until the next token. - Return rate-limit headers on **all** responses, not just 429s (`RateLimit-Limit`, `RateLimit-Remaining`, `RateLimit-Reset`, per the IETF draft, or the common `X-RateLimit-*`). Clients can then self-throttle. - Thundering herd: if 500 clients all get "retry in 12s," they all return at once. Document that clients should add jitter, and consider adding some server-side. - Use a structured error body with a stable error code. ## 6. Rollout and observability - Ship in **shadow mode** first: compute and log "would have limited" without enforcing. This shows how many real customers you'd break, which is often a surprise. - Metrics: limit hits per key/tier, limiter latency, Redis errors, fail-open count. - Plan for customer communication, since someone currently doing 300/min will start getting 429s. ## Where I'd start Redis-backed token bucket (or sliding window counter) via Lua script, limits looked up from plan config, fail-open with local fallback, IP limiting ahead of auth, standard headers, and a shadow-mode launch. ## Questions that would change the design 1. Is the limit a **billing/contractual guarantee** or just **backend protection**? Exactness matters much more for the former. 2. Do you already run Redis (or similar), and is the API multi-region? 3. Do you autoscale, or is it a fixed 6 servers? 4. Is the limit per key or per customer account? 5. Do you have endpoints with wildly different costs? Which of these do you want to dig into first?