Parallel Processing

Definition

Parallel processing means running many computations in the same instant on separate hardware, instead of one after another. It matters for AI because a GPU is built to perform the same operation on thousands of data elements at once, and that is exactly the shape of matrix multiplication — the operation that dominates neural network training and inference.

That single fit explains most of modern AI infrastructure. Training a large model is, underneath, an enormous pile of multiply-add operations with no dependencies between them, so they can all be computed simultaneously. A CPU works through them a few dozen at a time; an NVIDIA H100 has 16,896 arithmetic cores and does thousands at once. The model that would take a CPU months trains on a GPU cluster in days for the same reason a photocopier beats a scribe: the work was always parallel, and the hardware finally matches its shape.

Keep parallelism distinct from concurrency, which is often confused with it. Concurrency is about dealing with many things at once — interleaving tasks so none blocks the others, which a single core can do by rapidly switching between them. Parallelism is about doing many things at once, which requires many execution units physically running at the same time. A single-core machine can be concurrent; only multiple cores can be parallel.

How It Works

The hardware trick underneath GPU parallelism is SIMD — Single Instruction, Multiple Data. Rather than issuing one instruction per data element, a SIMD unit issues one instruction that a whole array of arithmetic units executes at once on different data. NVIDIA's variant is called SIMT (Single Instruction, Multiple Threads): the GPU runs threads in lock-step groups (a warp of 32 threads), all executing the same instruction on their own data in the same clock cycle. This is why a GPU has thousands of simple cores instead of a handful of fast, flexible ones — it trades per-core cleverness for raw throughput.

Matrix multiplication is the workload this was built for. Multiplying two N×N matrices is 2N³ floating-point operations — N³ multiply-adds, and every one of them is independent of the others. For a modest 8192×8192 multiply that is about 2 × 8192³ ≈ 1.1 trillion operations that can, in principle, all run at once. No operation waits on another's result, so there is nothing to serialize. On an H100, dedicated Tensor Cores (528 of them) do these multiply-accumulates in hardware, reaching roughly 989 dense FP16 TFLOPS — NVIDIA's headline "1,979 TFLOPS" figure is the same number doubled for structured sparsity, not the dense rate, and the two are not interchangeable.

Amdahl's law: the ceiling nobody expects

Here is what breaks, and it breaks the intuition that "more GPUs means proportionally faster." Almost no real program is 100% parallelizable — there is always setup, coordination, or a sequential dependency. Amdahl's law puts a hard number on what that costs. If p is the fraction of work that parallelizes, the speedup on N processors is:

speedup(N) = 1 / ((1 - p) + p/N)

Suppose 95% of a workload parallelizes perfectly (p = 0.95). Push N to infinity — infinite GPUs, zero cost per unit of parallel work — and the parallel term p/N vanishes, leaving 1 / (1 - 0.95) = 1/0.05 = 20x. That is the ceiling. The 5% that must run sequentially can never be sped up, so it alone caps the entire program at a 20x speedup no matter how much hardware you throw at it.

The approach to that ceiling is brutally fast. With N = 256 processors the same formula gives 1 / (0.05 + 0.95/256) ≈ 18.6x — you have spent 256x the hardware to get 18.6x the speed, already 93% of the theoretical maximum. Doubling again to 512 processors moves you only to ≈ 19.3x. The last stretch toward 20x costs unbounded hardware for almost nothing. This is why scaling a model to more devices eventually stops helping: past a point you are buying communication overhead, not speed, and the sequential fraction that was invisible on one GPU becomes the whole bill.

Types

Training a model too large for one GPU forces a choice about how to split the work, and there is a real, named three-way taxonomy for it. Frontier-scale training uses all three at once — "3D parallelism" — but they solve different problems and it is worth keeping them straight.

Data parallelism puts a complete copy of the model on every device and gives each device a different slice of the batch. Each computes gradients on its slice, then all devices average their gradients (an all-reduce communication step) so every copy stays identical. It is the simplest form and the default in frameworks like PyTorch's DistributedDataParallel and FSDP. Its limit is memory: it only works when a full copy of the model fits on one device.

Tensor parallelism (also called intra-layer model parallelism) splits a single layer's weight matrices across devices when one layer is too big to fit on one GPU. A large matrix multiply is partitioned column-wise or row-wise, each device computes its shard, and the shards are combined. This is the approach NVIDIA's Megatron-LM introduced for transformer layers. It removes the memory ceiling of data parallelism but demands very fast interconnects, because devices must communicate within every layer, on every forward and backward pass.

Pipeline parallelism puts different layers on different devices — device 1 holds layers 1-10, device 2 holds layers 11-20, and so on — and streams the batch through them like an assembly line. The problem it introduces is the "pipeline bubble": while device 1 processes the first micro-batch, devices 2 and 3 sit idle waiting for input. Google's GPipe and later schedules attack this by splitting the batch into many small micro-batches so all stages stay busy, shrinking the idle time.

Real-World Applications

Training large language models. Every frontier model is trained with combined data, tensor, and pipeline parallelism across thousands of GPUs. The specific split is an engineering decision: tensor parallelism is kept within a fast-interconnect node (the GPUs wired together with NVLink), because it is communication-heavy, while pipeline and data parallelism span across nodes over slower networking. Getting this mapping wrong — putting tensor parallelism across a slow network link — can make a cluster spend most of its time waiting rather than computing. This is the core work of distributed training.

Everyday deep learning. Far below frontier scale, ordinary training uses data parallelism through nn.DataParallel or DistributedDataParallel to spread a batch across the 2-8 GPUs in a single server, and vectorization to make each GPU process a whole batch of examples as one matrix operation rather than looping over them. Inference is parallel too: a served model batches many users' requests together so their forward passes run as a single large matrix multiply.

Beyond AI. The same principle drives scientific computing (climate and molecular simulations decompose a physical domain across processors), rendering (each frame or tile computed independently), and large-scale data processing frameworks like Spark that partition a dataset across a cluster. These predate deep learning and are the broader field of distributed computing; AI is the workload that made massively parallel hardware mainstream.

Key Concepts

• Speedup and efficiency: Speedup is sequential time divided by parallel time; efficiency is speedup divided by the number of processors. Efficiency near 1.0 means processors are well used; it falls as communication and Amdahl's sequential fraction take over.
• Amdahl's law vs Gustafson's law: Amdahl fixes the problem size and finds a speedup ceiling. Gustafson observes that in practice people grow the problem to fill available hardware — bigger models, bigger batches — so the parallel fraction rises with scale and the ceiling is less binding than Amdahl alone suggests. Both are true; they answer different questions.
• Communication overhead: The cost of moving data and synchronizing between devices. It grows with the number of devices and is the practical reason parallel scaling stops paying off, distinct from the theoretical Amdahl ceiling.
• Memory bandwidth: Matrix multiplication at scale is frequently limited not by arithmetic but by how fast weights can be fed from memory to the GPU's cores. A GPU can be arithmetically idle while waiting on memory — throughput without bandwidth to match is wasted.

Challenges

The central difficulty is that the two ceilings above are real and cannot be engineered away, only managed. Amdahl's sequential fraction and communication overhead both mean that beyond some device count, adding hardware slows the marginal return to near zero — so the hard question in large-scale training is not "can we add more GPUs" but "where does the payoff stop," and it stops sooner than budgets assume.

Load imbalance is the quiet killer. Parallel work finishes only when the slowest participant finishes, so one device with slightly more work, or one pipeline stage that is heavier than the others, stalls everyone else. A pipeline whose stages are unevenly sized wastes GPUs on the bubble; a data-parallel job with uneven batch sizes waits on its straggler at every all-reduce.

Debugging is genuinely harder than sequential code because bugs depend on timing. Race conditions and deadlocks appear only under specific interleavings and often vanish when you add logging (which changes the timing), making them maddening to reproduce. A parallel program can produce correct output on ten runs and a silently wrong number on the eleventh.

The memory wall increasingly dominates. As arithmetic throughput has raced ahead, the bottleneck for large matmuls has shifted to memory bandwidth — feeding the cores, not the cores themselves. Techniques like FlashAttention exist precisely to restructure a computation so it moves less data between memory levels, trading extra arithmetic (which is cheap and parallel) for fewer memory trips (which are the real constraint).

Code Example

This shows the SIMD/SIMT idea directly: the same operation applied to a whole array at once, with no Python loop over elements. On a GPU, the elements of the array are processed by different cores in parallel; the code you write is identical whether it runs on one core or thousands.

import numpy as np

# Sequential thinking: one element at a time (what we are NOT doing)
def elementwise_square_loop(x):
    out = np.empty_like(x)
    for i in range(len(x)):        # each iteration waits for the last
        out[i] = x[i] * x[i]
    return out

# Data-parallel thinking: one operation over the whole array
def elementwise_square_vectorized(x):
    return x * x                    # every element handled independently

data = np.arange(1_000_000, dtype=np.float32)
assert np.allclose(elementwise_square_loop(data),
                   elementwise_square_vectorized(data))

The two functions compute the same result, but the second expresses the work as a single parallel operation instead of a sequence of dependent steps. Moving from the loop to the vectorized form is the mental shift that lets the same code run on a GPU:

import torch

x = torch.arange(1_000_000, dtype=torch.float32, device="cuda")
y = x * x   # dispatched across the GPU's cores; the elements are independent

# Matrix multiplication is the same idea at scale: N^3 independent
# multiply-adds, all issued to the Tensor Cores at once.
a = torch.randn(8192, 8192, device="cuda")
b = torch.randn(8192, 8192, device="cuda")
c = a @ b   # ~1.1 trillion FLOPs, none of them waiting on another

The @ operator hides the parallelism, but that one line is where a GPU spends most of a training run — and where every idea in this article, from SIMT to Amdahl's law to the three kinds of model parallelism, is really about making that single operation, repeated billions of times, run as close to the hardware's peak as the sequential parts and the memory wall allow.