---
source: 'https://howaiworks.ai/glossary/concurrency'
section: glossary
title: Concurrency
description: >-
  Concurrency is structuring a program so independent tasks interleave. How it
  differs from parallelism, and why serving a model is a queueing problem.
tags:
  - parallel processing
  - Performance Optimization
  - Distributed Computing
  - AI Infrastructure
category: Infrastructure & Hardware
datePublished: '2025-08-20'
lastUpdated: '2026-07-22'
---

# Concurrency

> Concurrency is structuring a program so independent tasks interleave. How it differs from parallelism, and why serving a model is a queueing problem.

## Definition

Concurrency is a way of **structuring** a program so that several tasks are in progress at once,
each able to make headway whenever the others are stalled. It is not the same thing as doing
several things at once. The one-line distinction, which is what most people arrive here looking
for:

> **Concurrency is about structure. Parallelism is about execution.**
> Concurrency is *dealing with* many things at once; parallelism is *doing* many things at once.

That difference has a hard test attached. A single-core machine can be concurrent and **cannot** be
parallel — it has one execution unit, so at any given instant exactly one instruction is running,
yet it can still hold a thousand half-finished tasks and switch between them. Concurrency is a
property of how the work is *organised*; parallelism is a property of the *hardware it lands on*.
The same concurrent program becomes parallel, without a line changing, when you give it more cores.

The practical consequence is that concurrency is what you design and parallelism is what you buy.
Getting them confused produces two expensive mistakes in opposite directions: adding GPUs to a
service whose real problem is that it processes requests one at a time, and rewriting a
CPU-saturated numerical kernel with `async` when what it needed was
[parallel processing](https://howaiworks.ai/glossary/parallel-processing).

## How It Works

### Concurrency without parallelism, which is the case people find hardest

Take an [AI agent](https://howaiworks.ai/glossary/ai-agent) that has to make 20 tool calls — search a document store,
hit three APIs, call a model. Each one is a network round trip that takes about 800 ms, nearly all
of it spent waiting for a reply. Run them one after another and you wait `20 x 800 ms = 16
seconds`, during which your CPU is idle for roughly 15.99 of those seconds.

Now restructure it: a single thread issues all 20 requests, then parks itself and waits for
whichever reply arrives first. Total wall-clock time is a little over **800 ms** — a **20x**
speedup — on exactly one core, with no two instructions ever executing simultaneously. Nothing was
computed in parallel. What overlapped was the *waiting*. That is concurrency in its purest form,
and it is why Node.js, Python's `asyncio` and Go's goroutines exist.

### Parallelism without concurrency

The mirror case is a single matrix multiply. Multiplying two 4,096 x 4,096 matrices is
`2 x 4,096³ ≈ 137 billion` floating-point operations arranged as 16.8 million independent dot
products, none of which depends on any other. There is no interleaving to design, no task that
might block, nothing to schedule — one logical operation, spread across as much silicon as you can
give it. That is pure parallelism, and it is what a [GPU](https://howaiworks.ai/glossary/gpu-computing) is for. See
[matrix multiplication](https://howaiworks.ai/glossary/matrix-multiplication) for why this shape dominates neural
network compute.

### The two ways a task gets interrupted

Concurrent systems differ mainly in *who decides* when a task yields. With OS threads the
scheduler preempts you: your task can be suspended between any two instructions, which is why
shared data needs locks. With coroutines — `async`/`await`, goroutines, green threads — you yield
**cooperatively**, only at points you marked, which makes reasoning far easier but means a single
blocking call freezes everything. The classic bug in an AI service is exactly this: one synchronous
`requests.get()` inside an async handler, and a server that was serving 500 users concurrently
starts serving them one at a time.

### Serving a model is a queueing problem, not a compute problem

Here is where concurrency stops being a programming-language topic and starts being the thing that
decides your infrastructure bill. An [inference](https://howaiworks.ai/glossary/inference) server is a queue, and queues
obey **Little's Law**, which is the most useful piece of arithmetic on this page:

```
L = λ × W
```

The average number of requests in the system equals the arrival rate times the average time each
spends there. It holds for any stable queue — no assumptions about distributions, no fine print.
Work it: your service takes **5 seconds** to answer an average request and receives **40 requests
per second**. Then `L = 40 x 5 = 200` requests are inside your system at any moment. Not
"eventually" — right now, on average, permanently.

That single number sizes the system. If your fleet is eight replicas each holding a batch of 32,
you have **256 concurrent slots**, so 200 in flight puts you at **78% utilisation** and you are
fine. If you sized it at four replicas — 128 slots — you cannot hold 200, the queue grows without
bound, and no amount of faster hardware fixes it, because the deficit is structural.

### Why high utilisation is a trap

The second piece of queueing arithmetic explains a thing every operator eventually learns the hard
way. For a simple queue, the time a request spends in the system is

```
W = service time / (1 - utilisation)
```

That denominator goes to zero, so the curve has a **vertical asymptote**. With a 2-second service
time:

| Utilisation | Time in system | vs. service time |
|---|---|---|
| 50% | 4 s | 2x |
| 80% | 10 s | 5x |
| 90% | 20 s | 10x |
| 95% | 40 s | 20x |
| 99% | 200 s | 100x |

Read the last three rows. Moving from 90% to 95% utilisation is about **5.6% more traffic**, and it
**doubles** the time every user waits. Another 4.2% on top of that makes it **five times worse
again**. This is the formula for a single server and a many-slot fleet is gentler through the
middle of the range, but the asymptote is identical — every queue in the world blows up at 100%,
and the only question is how close you dared to run. It is why a fleet that looks 30% wasted on the
utilisation dashboard is often correctly sized, and why the request to "get the GPUs busier" is
sometimes a request to break the p99.

### Continuous batching: the scheduling structure that makes those slots real

Little's Law says you need 200 concurrent slots. Actually *holding* 200 is the hard part, because
generations have wildly different lengths and nobody knows a length in advance.

Under **static batching** you gather a cohort of B requests, run them together, and return when the
last one finishes. A slot is therefore held for the length of the *longest* sequence in the cohort,
not its own. So the fraction of slot-time doing real work is:

```
efficiency = mean output length / max output length
```

With a batch of 32 whose answers run from 50 to 2,000 tokens and average 400, that is
`400 / 2,000 = 20%`. Four fifths of your expensive concurrency is a finished sequence's corpse
occupying a slot.

**Continuous batching** — also called in-flight batching — fixes this by changing the *unit of
scheduling* from the request to the **iteration**. After every single forward pass the scheduler
re-evaluates the batch: any sequence that emitted its end-of-sequence [token](https://howaiworks.ai/glossary/token) is
evicted immediately, and a request waiting in the queue is admitted into the freed slot on the very
next step. The batch stops being a cohort and becomes a rolling population, so there is no
straggler for anyone to wait behind and the `mean/max` penalty disappears. The priced consequences
— cost per million tokens against batch size — are worked out on
[inference optimization](https://howaiworks.ai/glossary/inference-optimization); what matters here is that continuous
batching is a *concurrency* technique, not a compute one. It bought its throughput by never leaving
a slot empty.

### One knob, two customers

Batch size is where concurrency's central trade-off becomes concrete, and the trap is that the two
things it controls are wanted by **different people**.

The **operator** wants tokens per second per node, and raises the batch size to get it: during
generation the model's weights are read from memory once per step and reused by every sequence in
the batch, so more sequences means more output for the same bytes moved. (The arithmetic behind
"decode is memory-bound" — roughly two operations per byte read, against the hundreds a modern
accelerator needs to stay busy — is derived on [AI infrastructure](https://howaiworks.ai/glossary/ai-infrastructure) and
[memory wall](https://howaiworks.ai/glossary/memory-wall); it is the reason batching works at all.)

The **individual user** wants time-to-first-token and a fast stream, and every one of those extra
sequences makes both worse: more time queued before admission, and a longer, heavier forward pass
per token once admitted. Neither party is wrong. They are simply different customers of the same
knob, and "tune the batch size" is not a technical decision you can make without first deciding
which of them you are selling to.

## Real-World Applications

**vLLM and NVIDIA TensorRT-LLM are continuous batching shipped as products.** vLLM's PagedAttention
allocates the [KV cache](https://howaiworks.ai/glossary/kv-cache) in fixed-size blocks rather than one contiguous
reservation per sequence, so a request can join or leave without fragmenting memory — a memory
allocator built specifically so that the *concurrency* structure above is implementable. Without it
you must reserve worst-case context length per slot, and the number of slots you can afford
collapses.

**Prefill/decode disaggregation is a concurrency decision about interference.** Reading a prompt
and generating a token have opposite hardware limits, so a long prompt admitted into a running
batch stalls every user currently streaming. Serving stacks increasingly split them onto separate
pools of machines precisely so that one class of work cannot preempt the other — the same reasoning
that keeps a slow background job off the request thread of a web server, at
[data-centre](https://howaiworks.ai/glossary/ai-data-center) scale.

**Rate limits you meet as an API user are Little's Law facing outward.** A provider that caps you
on tokens per minute is capping `λ`; a cap on concurrent requests is capping `L` directly. Batch
endpoints, which return within hours rather than seconds, are the same product sold with `W`
relaxed by four orders of magnitude, which lets the scheduler place your work wherever the fleet
has a free slot.

**Agent frameworks are I/O-bound orchestrators.** A [RAG](https://howaiworks.ai/glossary/retrieval-augmented-generation)
or [multi-agent](https://howaiworks.ai/glossary/multi-agent-systems) pipeline spends the overwhelming majority of its
wall-clock time waiting on model calls it did not compute itself. Its scaling limit is how many
requests it can keep in flight, which is why these frameworks are written on event loops: an OS
thread reserves megabytes of stack to sit and wait, while a coroutine parked on the same wait costs
kilobytes. At ten thousand concurrent conversations that ratio decides whether the service fits on
one machine.

## Key Concepts

- **Little's Law (`L = λW`)**: the sizing tool. It tells you how many concurrent slots you need
  before you provision anything, and it is exact for any stable queue — which makes it one of the
  few numbers in systems engineering that never needs revising.
- **Utilisation is not a score to maximise**: because delay scales as `1/(1 - ρ)`, the last 10% of
  capacity costs more latency than the first 90% combined. Headroom is the product, not waste.
- **Race condition**: two tasks interleaving in an order nobody tested. In a serving stack the
  canonical one is the KV-cache block allocator — two schedulers both observe free blocks, both
  admit a request, and the node runs out of memory mid-generation, forcing it to preempt a
  half-finished sequence and recompute its cache from scratch.
- **Deadlock**: everyone waiting for everyone. In tensor-parallel serving every rank must enter
  each collective operation in the same order; if one rank's scheduler takes a different branch and
  skips a step, the other ranks block forever in the collective and the whole replica hangs with
  the GPUs at 0% and no error message.
- **Backpressure**: the mechanism for refusing work you cannot hold. A server that accepts requests
  faster than it retires them converts a throughput problem into an out-of-memory crash; rejecting
  early with a 429 is the correct behaviour, not a failure.

## Challenges

**Tail latency is where concurrency is actually judged, and averages hide it.** Little's Law gives
you a mean, and a mean is exactly the statistic that looks healthy while your p99 is on fire. A
fleet sized for its average arrival rate will meet its median target and miss its 99th percentile
by an order of magnitude the first time traffic arrives in a burst, because during the burst
utilisation is momentarily near 1 and the `1/(1 - ρ)` curve does the rest. Size for the peak you
intend to survive, not the mean you measured.

**The failures are non-deterministic, which breaks normal debugging.** A race condition that
appears once every ten thousand requests will not reproduce under a debugger, because attaching one
changes the timing that caused it. This is the fundamental tax on concurrent code and the reason
languages have spent a decade moving toward structures — message passing, ownership rules,
structured concurrency — that make certain races impossible to express rather than merely rare.

**More concurrency is not monotonically better.** Every additional in-flight request consumes KV
cache, and cache is the variable cost that eventually dominates a decode step. Past that point
raising the batch buys you almost no extra throughput while continuing to degrade every user's
latency — a knob that has stopped paying and not stopped costing. The crossover is computable for a
given model and node, and it is worked through on
[inference optimization](https://howaiworks.ai/glossary/inference-optimization).

**Concurrency's guarantees stop at the machine boundary.** A lock protects shared memory inside one
process; across a fleet there is no shared memory, clocks disagree, and messages arrive twice or
never — at which point this becomes a [distributed computing](https://howaiworks.ai/glossary/distributed-computing)
problem with a harder set of tools.

## Code Example

Concurrency without parallelism, made measurable. Both functions below run on a single thread and
do identical work; only the structure differs.

```python
import asyncio
import time

async def call_model(client, prompt: str) -> str:
    """Stands in for an API call: ~800 ms of waiting, ~0 ms of computing."""
    await asyncio.sleep(0.8)
    return f"answer to {prompt}"

async def sequential(client, prompts: list[str]) -> list[str]:
    # Structured as one task. Total time = sum of the waits.
    return [await call_model(client, p) for p in prompts]

async def concurrent(client, prompts: list[str]) -> list[str]:
    # Structured as N tasks. Total time = the longest single wait.
    tasks = [call_model(client, p) for p in prompts]
    return await asyncio.gather(*tasks)

async def main() -> None:
    prompts = [f"question {i}" for i in range(20)]

    for label, runner in (("sequential", sequential), ("concurrent", concurrent)):
        start = time.perf_counter()
        await runner(None, prompts)
        print(f"{label}: {time.perf_counter() - start:.2f}s")

asyncio.run(main())
# sequential: 16.01s
# concurrent: 0.80s
```

One core, one thread, no parallelism anywhere — and a 20x difference that comes entirely from how
the work was arranged. Note that `asyncio.gather` is the concurrency here; `await` on its own is
just a sequential call with extra syntax, which is the single most common misreading of async code.
In production you would also bound it with an `asyncio.Semaphore`: firing all 20 at once is fine,
but the same code with 20,000 prompts is a self-inflicted denial-of-service on whatever you are
calling, and Little's Law applies to the service at the other end too.

## Frequently Asked Questions

### What's the difference between concurrency and parallelism?

Concurrency is about structure, parallelism is about execution. A concurrent program is organised so independent tasks can interleave in any order; a parallel program physically runs tasks at the same instant on different hardware. A single-core machine can be concurrent and cannot be parallel.

### Can you have concurrency without parallelism?

Yes, and it is the most common case in practice. A single-threaded event loop issuing 20 API calls that each wait 800 ms finishes in about 800 ms instead of 16 seconds, without ever executing two instructions at the same instant. The waiting overlapped; the computing never did.

### Why does a server get dramatically slower near full utilisation?

Queueing delay scales roughly as 1/(1 - utilisation), so it has a vertical asymptote. Going from 90% to 95% utilisation is about 5% more traffic and roughly doubles the time a request spends in the system. This is why serving fleets are deliberately run with headroom.

### What is continuous batching in LLM serving?

It reschedules the batch after every forward pass instead of every request. A sequence that finishes is evicted immediately and a queued request takes its slot on the next step, so no slot sits idle waiting for the longest generation in a cohort to finish.

### Why is async I/O the right model for AI applications?

An agent or RAG service spends nearly all of its wall-clock time waiting on model calls, not computing. The scarce resource is in-flight requests rather than CPU, and a coroutine costs kilobytes to keep waiting where an OS thread reserves megabytes of stack.

## Related

### Related terms

- [Parallel Processing](https://howaiworks.ai/glossary/parallel-processing)
- [Inference Optimization](https://howaiworks.ai/glossary/inference-optimization)
- [KV Cache](https://howaiworks.ai/glossary/kv-cache)
- [Memory Wall](https://howaiworks.ai/glossary/memory-wall)
- [Distributed Computing](https://howaiworks.ai/glossary/distributed-computing)
- [Consensus Algorithm](https://howaiworks.ai/glossary/consensus-algorithm)

---

Source: https://howaiworks.ai/glossary/concurrency — HowAIWorks.ai