Concurrency

Definition

Concurrency is a way of structuring a program so that several tasks are in progress at once, each able to make headway whenever the others are stalled. It is not the same thing as doing several things at once. The one-line distinction, which is what most people arrive here looking for:

Concurrency is about structure. Parallelism is about execution. Concurrency is dealing with many things at once; parallelism is doing many things at once.

That difference has a hard test attached. A single-core machine can be concurrent and cannot be parallel — it has one execution unit, so at any given instant exactly one instruction is running, yet it can still hold a thousand half-finished tasks and switch between them. Concurrency is a property of how the work is organised; parallelism is a property of the hardware it lands on. The same concurrent program becomes parallel, without a line changing, when you give it more cores.

The practical consequence is that concurrency is what you design and parallelism is what you buy. Getting them confused produces two expensive mistakes in opposite directions: adding GPUs to a service whose real problem is that it processes requests one at a time, and rewriting a CPU-saturated numerical kernel with async when what it needed was parallel processing.

How It Works

Concurrency without parallelism, which is the case people find hardest

Take an AI agent that has to make 20 tool calls — search a document store, hit three APIs, call a model. Each one is a network round trip that takes about 800 ms, nearly all of it spent waiting for a reply. Run them one after another and you wait 20 x 800 ms = 16 seconds, during which your CPU is idle for roughly 15.99 of those seconds.

Now restructure it: a single thread issues all 20 requests, then parks itself and waits for whichever reply arrives first. Total wall-clock time is a little over 800 ms — a 20x speedup — on exactly one core, with no two instructions ever executing simultaneously. Nothing was computed in parallel. What overlapped was the waiting. That is concurrency in its purest form, and it is why Node.js, Python's asyncio and Go's goroutines exist.

Parallelism without concurrency

The mirror case is a single matrix multiply. Multiplying two 4,096 x 4,096 matrices is 2 x 4,096³ ≈ 137 billion floating-point operations arranged as 16.8 million independent dot products, none of which depends on any other. There is no interleaving to design, no task that might block, nothing to schedule — one logical operation, spread across as much silicon as you can give it. That is pure parallelism, and it is what a GPU is for. See matrix multiplication for why this shape dominates neural network compute.

The two ways a task gets interrupted

Concurrent systems differ mainly in who decides when a task yields. With OS threads the scheduler preempts you: your task can be suspended between any two instructions, which is why shared data needs locks. With coroutines — async/await, goroutines, green threads — you yield cooperatively, only at points you marked, which makes reasoning far easier but means a single blocking call freezes everything. The classic bug in an AI service is exactly this: one synchronous requests.get() inside an async handler, and a server that was serving 500 users concurrently starts serving them one at a time.

Serving a model is a queueing problem, not a compute problem

Here is where concurrency stops being a programming-language topic and starts being the thing that decides your infrastructure bill. An inference server is a queue, and queues obey Little's Law, which is the most useful piece of arithmetic on this page:

L = λ × W

The average number of requests in the system equals the arrival rate times the average time each spends there. It holds for any stable queue — no assumptions about distributions, no fine print. Work it: your service takes 5 seconds to answer an average request and receives 40 requests per second. Then L = 40 x 5 = 200 requests are inside your system at any moment. Not "eventually" — right now, on average, permanently.

That single number sizes the system. If your fleet is eight replicas each holding a batch of 32, you have 256 concurrent slots, so 200 in flight puts you at 78% utilisation and you are fine. If you sized it at four replicas — 128 slots — you cannot hold 200, the queue grows without bound, and no amount of faster hardware fixes it, because the deficit is structural.

Why high utilisation is a trap

The second piece of queueing arithmetic explains a thing every operator eventually learns the hard way. For a simple queue, the time a request spends in the system is

W = service time / (1 - utilisation)

That denominator goes to zero, so the curve has a vertical asymptote. With a 2-second service time:

Read the last three rows. Moving from 90% to 95% utilisation is about 5.6% more traffic, and it doubles the time every user waits. Another 4.2% on top of that makes it five times worse again. This is the formula for a single server and a many-slot fleet is gentler through the middle of the range, but the asymptote is identical — every queue in the world blows up at 100%, and the only question is how close you dared to run. It is why a fleet that looks 30% wasted on the utilisation dashboard is often correctly sized, and why the request to "get the GPUs busier" is sometimes a request to break the p99.

Continuous batching: the scheduling structure that makes those slots real

Little's Law says you need 200 concurrent slots. Actually holding 200 is the hard part, because generations have wildly different lengths and nobody knows a length in advance.

Under static batching you gather a cohort of B requests, run them together, and return when the last one finishes. A slot is therefore held for the length of the longest sequence in the cohort, not its own. So the fraction of slot-time doing real work is:

efficiency = mean output length / max output length

With a batch of 32 whose answers run from 50 to 2,000 tokens and average 400, that is 400 / 2,000 = 20%. Four fifths of your expensive concurrency is a finished sequence's corpse occupying a slot.

Continuous batching — also called in-flight batching — fixes this by changing the unit of scheduling from the request to the iteration. After every single forward pass the scheduler re-evaluates the batch: any sequence that emitted its end-of-sequence token is evicted immediately, and a request waiting in the queue is admitted into the freed slot on the very next step. The batch stops being a cohort and becomes a rolling population, so there is no straggler for anyone to wait behind and the mean/max penalty disappears. The priced consequences — cost per million tokens against batch size — are worked out on inference optimization; what matters here is that continuous batching is a concurrency technique, not a compute one. It bought its throughput by never leaving a slot empty.

One knob, two customers

Batch size is where concurrency's central trade-off becomes concrete, and the trap is that the two things it controls are wanted by different people.

The operator wants tokens per second per node, and raises the batch size to get it: during generation the model's weights are read from memory once per step and reused by every sequence in the batch, so more sequences means more output for the same bytes moved. (The arithmetic behind "decode is memory-bound" — roughly two operations per byte read, against the hundreds a modern accelerator needs to stay busy — is derived on AI infrastructure and memory wall; it is the reason batching works at all.)

The individual user wants time-to-first-token and a fast stream, and every one of those extra sequences makes both worse: more time queued before admission, and a longer, heavier forward pass per token once admitted. Neither party is wrong. They are simply different customers of the same knob, and "tune the batch size" is not a technical decision you can make without first deciding which of them you are selling to.

Real-World Applications

vLLM and NVIDIA TensorRT-LLM are continuous batching shipped as products. vLLM's PagedAttention allocates the KV cache in fixed-size blocks rather than one contiguous reservation per sequence, so a request can join or leave without fragmenting memory — a memory allocator built specifically so that the concurrency structure above is implementable. Without it you must reserve worst-case context length per slot, and the number of slots you can afford collapses.

Prefill/decode disaggregation is a concurrency decision about interference. Reading a prompt and generating a token have opposite hardware limits, so a long prompt admitted into a running batch stalls every user currently streaming. Serving stacks increasingly split them onto separate pools of machines precisely so that one class of work cannot preempt the other — the same reasoning that keeps a slow background job off the request thread of a web server, at data-centre scale.

Rate limits you meet as an API user are Little's Law facing outward. A provider that caps you on tokens per minute is capping λ; a cap on concurrent requests is capping L directly. Batch endpoints, which return within hours rather than seconds, are the same product sold with W relaxed by four orders of magnitude, which lets the scheduler place your work wherever the fleet has a free slot.

Agent frameworks are I/O-bound orchestrators. A RAG or multi-agent pipeline spends the overwhelming majority of its wall-clock time waiting on model calls it did not compute itself. Its scaling limit is how many requests it can keep in flight, which is why these frameworks are written on event loops: an OS thread reserves megabytes of stack to sit and wait, while a coroutine parked on the same wait costs kilobytes. At ten thousand concurrent conversations that ratio decides whether the service fits on one machine.

Key Concepts

• Little's Law (L = λW): the sizing tool. It tells you how many concurrent slots you need before you provision anything, and it is exact for any stable queue — which makes it one of the few numbers in systems engineering that never needs revising.
• Utilisation is not a score to maximise: because delay scales as 1/(1 - ρ), the last 10% of capacity costs more latency than the first 90% combined. Headroom is the product, not waste.
• Race condition: two tasks interleaving in an order nobody tested. In a serving stack the canonical one is the KV-cache block allocator — two schedulers both observe free blocks, both admit a request, and the node runs out of memory mid-generation, forcing it to preempt a half-finished sequence and recompute its cache from scratch.
• Deadlock: everyone waiting for everyone. In tensor-parallel serving every rank must enter each collective operation in the same order; if one rank's scheduler takes a different branch and skips a step, the other ranks block forever in the collective and the whole replica hangs with the GPUs at 0% and no error message.
• Backpressure: the mechanism for refusing work you cannot hold. A server that accepts requests faster than it retires them converts a throughput problem into an out-of-memory crash; rejecting early with a 429 is the correct behaviour, not a failure.

Challenges

Tail latency is where concurrency is actually judged, and averages hide it. Little's Law gives you a mean, and a mean is exactly the statistic that looks healthy while your p99 is on fire. A fleet sized for its average arrival rate will meet its median target and miss its 99th percentile by an order of magnitude the first time traffic arrives in a burst, because during the burst utilisation is momentarily near 1 and the 1/(1 - ρ) curve does the rest. Size for the peak you intend to survive, not the mean you measured.

The failures are non-deterministic, which breaks normal debugging. A race condition that appears once every ten thousand requests will not reproduce under a debugger, because attaching one changes the timing that caused it. This is the fundamental tax on concurrent code and the reason languages have spent a decade moving toward structures — message passing, ownership rules, structured concurrency — that make certain races impossible to express rather than merely rare.

More concurrency is not monotonically better. Every additional in-flight request consumes KV cache, and cache is the variable cost that eventually dominates a decode step. Past that point raising the batch buys you almost no extra throughput while continuing to degrade every user's latency — a knob that has stopped paying and not stopped costing. The crossover is computable for a given model and node, and it is worked through on inference optimization.

Concurrency's guarantees stop at the machine boundary. A lock protects shared memory inside one process; across a fleet there is no shared memory, clocks disagree, and messages arrive twice or never — at which point this becomes a distributed computing problem with a harder set of tools.

Code Example

Concurrency without parallelism, made measurable. Both functions below run on a single thread and do identical work; only the structure differs.

import asyncio
import time

async def call_model(client, prompt: str) -> str:
    """Stands in for an API call: ~800 ms of waiting, ~0 ms of computing."""
    await asyncio.sleep(0.8)
    return f"answer to {prompt}"

async def sequential(client, prompts: list[str]) -> list[str]:
    # Structured as one task. Total time = sum of the waits.
    return [await call_model(client, p) for p in prompts]

async def concurrent(client, prompts: list[str]) -> list[str]:
    # Structured as N tasks. Total time = the longest single wait.
    tasks = [call_model(client, p) for p in prompts]
    return await asyncio.gather(*tasks)

async def main() -> None:
    prompts = [f"question {i}" for i in range(20)]

    for label, runner in (("sequential", sequential), ("concurrent", concurrent)):
        start = time.perf_counter()
        await runner(None, prompts)
        print(f"{label}: {time.perf_counter() - start:.2f}s")

asyncio.run(main())
# sequential: 16.01s
# concurrent: 0.80s

One core, one thread, no parallelism anywhere — and a 20x difference that comes entirely from how the work was arranged. Note that asyncio.gather is the concurrency here; await on its own is just a sequential call with extra syntax, which is the single most common misreading of async code. In production you would also bound it with an asyncio.Semaphore: firing all 20 at once is fine, but the same code with 20,000 prompts is a self-inflicted denial-of-service on whatever you are calling, and Little's Law applies to the service at the other end too.