Rate Limiting and Backpressure Between Services

A service has finite capacity. Callers can send requests faster than it can process them because of traffic spikes, bugs, retry storms, or one heavy client. Rate limiting restricts how many requests a caller may send in a period. Backpressure is any mechanism by which a slow consumer signals a faster producer to reduce its sending rate. Between microservices, both are needed to keep overload from spreading.

Rate limiting between services

Rate limiting between internal services differs from limiting public API clients:

• Callers are known and authenticated, so limits can be set per calling service.
• Limits protect shared dependencies (databases, downstream services) from a misbehaving neighbor.
• Legitimate traffic is bursty, so limits should allow controlled bursts.

The algorithms (token bucket, leaky bucket, fixed and sliding windows) are described in rate limiting algorithms. This article covers where limits are placed and how limit signals travel.

Where to enforce

Server-side limiting is authoritative: it works even if clients are buggy. Client-side throttling reduces wasted network calls and lets a caller degrade gracefully instead of receiving errors. Systems typically use both: the server enforces, and well-behaved clients slow down when they receive rejection signals.

Adaptive client-side throttling (described in the Google SRE book) lets a client track its recent requests and the requests the server accepted. It rejects new requests locally with probability max(0, (requests - K * accepts) / (requests + 1)), where K is a multiplier such as 2. While the server accepts nearly everything, the probability is zero. As rejections dominate, the client sheds an increasing share of its own calls without contacting the server, which reduces load on an overloaded backend.

Local versus distributed limits

• Local limit: each instance enforces its own counter. Simple and fast. The effective total limit is the per-instance limit multiplied by the number of instances, which varies as instances scale. For example, a target of 100 requests per second split as 25 per second across 4 instances rejects requests on an instance receiving 40 per second even though the other instances have spare capacity, and the split must be recomputed when the instance count changes.
• Distributed limit: instances share counters through a store such as Redis or through a dedicated rate limit service. The limit is accurate globally, at the cost of an extra network call per request and a new dependency that must itself fail open or closed.

When the rate limit store is unreachable, the limiter must choose: fail open (allow traffic, risking overload) or fail closed (reject traffic, causing an outage). Internal protection limits usually fail open with a local fallback limit.

Quotas and fairness

A quota is a limit over a longer period, such as requests per day or per month, often tied to a plan or a budget. A rate limit controls short-term speed, and a quota controls total usage.

To keep one caller from taking all capacity:

• Assign per-caller limits or shares (for example each calling service gets a reserved share of capacity).
• Use weighted fair queueing or priority classes so critical callers are served first (see bulkhead, fallback and load shedding).
• Charge requests by cost instead of by count when requests differ greatly in expense.

Signaling limits to callers

The server should tell the caller that it was throttled and when to retry.

• HTTP 429 Too Many Requests indicates the client exceeded a limit. 503 Service Unavailable indicates server-side overload or unavailability.
• The Retry-After header gives seconds or a date after which the client may retry.
• Many APIs also send rate limit headers such as RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset (defined in an IETF draft whose header names have changed between revisions, with older X-RateLimit-* variants in wide use).
• In gRPC, RESOURCE_EXHAUSTED and UNAVAILABLE status codes serve this role.

Clients must react correctly: stop or slow sending, wait at least the indicated time, and apply backoff with jitter to retries. A client that retries a 429 immediately makes the overload worse.

Backpressure

Backpressure propagates the consumer's capacity limit to the producer. Without it, the excess load accumulates in an unbounded buffer, which grows until memory is exhausted or latency becomes unbounded.

flowchart LR
    P["Producer"] -->|"requests"| B["Bounded buffer"]
    B --> C["Consumer"]
    C -.->|"slow: buffer full"| B
    B -.->|"signal: slow down or reject"| P

Options when a consumer cannot keep up:

1. Block the producer (bounded buffer, blocking send). Works inside one process; blocked threads also hold resources.
2. Reject new work with an error (429, 503). The producer must handle the rejection.
3. Drop work, oldest or newest, when data loss is acceptable (metrics, telemetry).
4. Slow the producer through a protocol-level mechanism (windowing, credits).

Backpressure in synchronous calls

In request/response systems, a bounded number of in-flight requests (concurrency limit) at each service is the main mechanism. When the limit is reached, the service rejects new requests, and the caller receives an error instead of waiting in an unbounded queue. The caller then rejects or degrades its own work, and the signal propagates up the call chain to the edge, where the user or client receives 503 or 429.

Protocol-level flow control exists at lower layers:

• TCP uses a receive window: a receiver that is slow advertises a smaller window, and the sender stops.
• HTTP/2 has per-stream and per-connection flow-control windows, and gRPC exposes them for streaming calls.
• Reactive Streams define a request(n) protocol: the subscriber tells the publisher how many items it can accept, and the publisher sends no more than that.

Backpressure in asynchronous systems

Queues decouple producer and consumer speed, but they do not remove the limit. A queue that grows continuously means consumers are slower than producers on average.

• Pull-based consumption (Kafka consumers poll, SQS consumers receive) gives natural backpressure: the consumer takes messages only when it has capacity. Producers are not slowed, and lag accumulates in the broker, which is durable and can absorb bursts.
• Push-based delivery (for example RabbitMQ deliveries) needs a bound on unacknowledged messages, which RabbitMQ provides as consumer prefetch (QoS) limits.
• Bounded queues can reject or block producers when full.
• Monitoring consumer lag or queue depth and scaling consumers, or throttling producers when lag exceeds a threshold, closes the loop.

See backpressure, retries, and dead-letter queues for the messaging side.

Combined behavior under overload

sequenceDiagram
    participant C as Caller
    participant S as Service
    C->>S: request
    S-->>C: 429 Retry-After 2
    Note over C: waits at least 2 s, plus jitter
    C->>S: retry
    S-->>C: 200

The full chain: the server limits concurrency and rate, rejects the excess quickly, and reports when to retry. The client honors the signal, backs off, and uses a circuit breaker so that sustained rejection stops requests altogether. Each layer bounds its queues.

Trade-offs

• Strict limits protect the service but reject legitimate bursts. Burst allowance (bucket size) trades protection for smoothness.
• Client-side throttling depends on client cooperation. Server-side limits cost server work to reject each request, so rejection must be cheap and happen early.
• Unbounded queues hide overload until failure. Bounded queues surface it immediately as rejections.
• Distributed limits are accurate but add latency and a dependency. Local limits are cheap but approximate.

Interview questions

• What is the difference between rate limiting and backpressure? Rate limiting enforces a policy on how much a caller may send. Backpressure is the feedback from a slow consumer that makes the producer reduce its rate. Rate limiting is policy-based, and backpressure is capacity-based.
• Why should an overloaded service return 429 or 503 with Retry-After instead of queueing? Queueing increases latency for all requests and consumes memory. A fast rejection with a retry hint keeps admitted requests fast and tells the caller when to try again.
• What are the options for a producer when a consumer is slower? Block the producer, reject or drop work, buffer in a bounded queue, or slow the producer through credits or windows. Unbounded buffering only delays the failure.
• How do pull-based consumers provide backpressure? The consumer fetches messages only when it can process them, so the broker holds the backlog. Lag measures how far the consumer is behind.
• Should a rate limiter fail open or closed when its store is down? For protective internal limits, usually open with a local fallback limit, to avoid turning a limiter failure into an outage. For limits tied to billing or security, closed may be required.