What 'Traffic' Actually Means at the Backend and API Layer

"Traffic" at the backend and API layer isn't one measurement. It's at least three distinct ones, and a server can be excellent at handling one while struggling badly with another. Requests per second measures how fast discrete request-response cycles complete: a speed metric, relevant to CPU-bound or short-lived work. Concurrent connections measures how many sockets or sessions are open at once, regardless of whether they're actively sending data: a resource-holding metric, relevant to memory, file descriptors, and connection-table state. Bandwidth measures raw data volume in bytes per second, distinct from request count entirely. Treating these as interchangeable is a common source of capacity-planning mistakes, because the thing that actually limits a server depends on which of the three is being stressed.

Why the distinction matters in practice

The clearest historical framing of this is the C10k problem, a term coined by Dan Kegel in 1999 to describe the challenge of serving 10,000 simultaneous clients from one server. The core insight, still accurate today: handling many concurrent connections is a genuinely different engineering problem from handling many requests per second. The latter requires high throughput. The former doesn't need to be fast at all; it requires efficient scheduling of connections that may sit mostly idle. A server architecture can be excellent at one and poor at the other. An event-driven server holds many idle connections cheaply, while a process- or thread-per-connection server exhausts memory at far lower concurrency, even if its raw throughput on active connections is fine.

This becomes concrete with long-lived connections: WebSockets and Server-Sent Events (SSE) hold a single connection open indefinitely, rather than the connect-request-respond-disconnect cycle HTTP/1.1 was originally designed around. WebSocket, standardized as RFC 6455, establishes one persistent, full-duplex TCP connection via an HTTP Upgrade handshake and keeps it open for framed messages in both directions. A server built to handle request-per-second throughput well can still struggle here, because the relevant resource isn't requests completing quickly, it's how many sockets can be held open simultaneously without exhausting memory. A practical operational detail worth knowing: browsers commonly cap concurrent HTTP/1.1 connections per origin at six, which becomes a real constraint for SSE specifically if a visitor has multiple tabs open against the same origin.

A related, useful reframe for long-lived-connection systems specifically: connection churn rate (connections opening and closing per second) is often a more meaningful health signal than raw concurrent-connection count. A stable set of 200,000 persistent connections strains a server far less than 20,000 connections that reconnect every 30 seconds, because each new connection carries its own setup cost regardless of how many are open at any given instant.

Relating the two with Little's Law

Queueing theory offers a formal way to tie throughput and concurrency together, rather than treating them as unrelated numbers. Little's Law, from operations research, states that average concurrency (L) equals arrival rate (λ) multiplied by the average time each item spends in the system (W): L = λW. Applied to a server, this is the formal justification for why requests per second and concurrent connections aren't interchangeable: they're related through latency. Increase how long each request takes to handle, and the number of concurrent connections needed to sustain the same throughput rises proportionally, even if the arrival rate of new requests never changes. It's a standard tool for reasoning about how a latency regression elsewhere in a system inflates the concurrency a server has to sustain, well before that concurrency shows up as an obvious problem on its own.

Applying it

• When diagnosing a server that's struggling under load, identify which of the three units is actually being stressed (request rate, concurrent connections, or bandwidth) before assuming the fix. Each has a different remedy.
• For any interface using long-lived connections (WebSockets, SSE, streaming responses), plan capacity around concurrent-connection count and churn rate specifically, not requests per second. The two are not proxies for each other in this case.
• Use Little's Law as a sanity check when a service's latency changes: a slower average response time means more concurrent load is needed to sustain the same throughput, which can turn a latency regression into a capacity problem even without any increase in incoming request volume.
• Treat "handles high RPS" and "handles high concurrency" as two separate claims about a system, each needing its own verification, not one general notion of "can handle traffic."