---
source: 'https://howaiworks.ai/glossary/parallel-processing'
section: glossary
title: Parallel Processing
description: >-
  Running many computations at the same instant across many processing units. In
  AI it makes matrix multiplication fast, which is why GPUs train models.
tags:
  - parallel processing
  - GPU
  - CUDA
  - Distributed Computing
  - Deep Learning
  - Performance Optimization
category: Infrastructure & Hardware
datePublished: '2025-08-17'
lastUpdated: '2026-07-24'
---

# Parallel Processing

> Running many computations at the same instant across many processing units. In AI it makes matrix multiplication fast, which is why GPUs train models.

## Definition

Parallel processing means running many computations in the *same instant* on separate hardware, instead of one after another. It matters for AI because a GPU is built to perform the same operation on thousands of data elements at once, and that is exactly the shape of [matrix multiplication](https://howaiworks.ai/glossary/matrix-multiplication) — the operation that dominates neural network training and inference.

That single fit explains most of modern AI infrastructure. Training a large model is, underneath, an enormous pile of multiply-add operations with no dependencies between them, so they can all be computed simultaneously. A CPU works through them a few dozen at a time; an NVIDIA H100 has 16,896 arithmetic cores and does thousands at once. The model that would take a CPU months trains on a GPU cluster in days for the same reason a photocopier beats a scribe: the work was always parallel, and the hardware finally matches its shape.

Keep parallelism distinct from [concurrency](https://howaiworks.ai/glossary/concurrency), which is often confused with it. Concurrency is about *dealing with* many things at once — interleaving tasks so none blocks the others, which a single core can do by rapidly switching between them. Parallelism is about *doing* many things at once, which requires many execution units physically running at the same time. A single-core machine can be concurrent; only multiple cores can be parallel.

## How It Works

The hardware trick underneath GPU parallelism is **SIMD** — Single Instruction, Multiple Data. Rather than issuing one instruction per data element, a SIMD unit issues one instruction that a whole array of arithmetic units executes at once on different data. NVIDIA's variant is called **SIMT** (Single Instruction, Multiple Threads): the GPU runs threads in lock-step groups (a *warp* of 32 threads), all executing the same instruction on their own data in the same clock cycle. This is why a GPU has thousands of simple cores instead of a handful of fast, flexible ones — it trades per-core cleverness for raw throughput.

Matrix multiplication is the workload this was built for. Multiplying two N×N matrices is `2N³` floating-point operations — `N³` multiply-adds, and every one of them is independent of the others. For a modest 8192×8192 multiply that is about `2 × 8192³ ≈ 1.1 trillion` operations that can, in principle, all run at once. No operation waits on another's result, so there is nothing to serialize. On an H100, dedicated **Tensor Cores** (528 of them) do these multiply-accumulates in hardware, reaching roughly 989 dense FP16 TFLOPS — NVIDIA's headline "1,979 TFLOPS" figure is the same number doubled for structured sparsity, not the dense rate, and the two are not interchangeable.

### Amdahl's law: the ceiling nobody expects

Here is what breaks, and it breaks the intuition that "more GPUs means proportionally faster." Almost no real program is 100% parallelizable — there is always setup, coordination, or a sequential dependency. **Amdahl's law** puts a hard number on what that costs. If `p` is the fraction of work that parallelizes, the speedup on `N` processors is:

```
speedup(N) = 1 / ((1 - p) + p/N)
```

Suppose 95% of a workload parallelizes perfectly (`p = 0.95`). Push `N` to infinity — infinite GPUs, zero cost per unit of parallel work — and the parallel term `p/N` vanishes, leaving `1 / (1 - 0.95) = 1/0.05 = 20x`. That is the *ceiling*. The 5% that must run sequentially can never be sped up, so it alone caps the entire program at a 20x speedup no matter how much hardware you throw at it.

The approach to that ceiling is brutally fast. With `N = 256` processors the same formula gives `1 / (0.05 + 0.95/256) ≈ 18.6x` — you have spent 256x the hardware to get 18.6x the speed, already 93% of the theoretical maximum. Doubling again to 512 processors moves you only to `≈ 19.3x`. The last stretch toward 20x costs unbounded hardware for almost nothing. This is why scaling a model to more devices eventually stops helping: past a point you are buying communication overhead, not speed, and the sequential fraction that was invisible on one GPU becomes the whole bill.

## Types

Training a model too large for one GPU forces a choice about *how* to split the work, and there is a real, named three-way taxonomy for it. Frontier-scale training uses all three at once — "3D parallelism" — but they solve different problems and it is worth keeping them straight.

**Data parallelism** puts a complete copy of the model on every device and gives each device a different slice of the batch. Each computes gradients on its slice, then all devices average their gradients (an *all-reduce* communication step) so every copy stays identical. It is the simplest form and the default in frameworks like PyTorch's DistributedDataParallel and FSDP. Its limit is memory: it only works when a full copy of the model fits on one device.

**Tensor parallelism** (also called intra-layer model parallelism) splits a single layer's weight matrices *across* devices when one layer is too big to fit on one GPU. A large matrix multiply is partitioned column-wise or row-wise, each device computes its shard, and the shards are combined. This is the approach NVIDIA's Megatron-LM introduced for transformer layers. It removes the memory ceiling of data parallelism but demands very fast interconnects, because devices must communicate *within* every layer, on every forward and backward pass.

**Pipeline parallelism** puts different *layers* on different devices — device 1 holds layers 1-10, device 2 holds layers 11-20, and so on — and streams the batch through them like an assembly line. The problem it introduces is the "pipeline bubble": while device 1 processes the first micro-batch, devices 2 and 3 sit idle waiting for input. Google's GPipe and later schedules attack this by splitting the batch into many small micro-batches so all stages stay busy, shrinking the idle time.

## Real-World Applications

**Training large language models.** Every frontier model is trained with combined data, tensor, and pipeline parallelism across thousands of GPUs. The specific split is an engineering decision: tensor parallelism is kept *within* a fast-interconnect node (the GPUs wired together with NVLink), because it is communication-heavy, while pipeline and data parallelism span *across* nodes over slower networking. Getting this mapping wrong — putting tensor parallelism across a slow network link — can make a cluster spend most of its time waiting rather than computing. This is the core work of [distributed training](https://howaiworks.ai/glossary/distributed-training).

**Everyday deep learning.** Far below frontier scale, ordinary training uses data parallelism through `nn.DataParallel` or DistributedDataParallel to spread a batch across the 2-8 GPUs in a single server, and [vectorization](https://howaiworks.ai/glossary/vectorization) to make each GPU process a whole batch of examples as one matrix operation rather than looping over them. Inference is parallel too: a served model batches many users' requests together so their forward passes run as a single large matrix multiply.

**Beyond AI.** The same principle drives scientific computing (climate and molecular simulations decompose a physical domain across processors), rendering (each frame or tile computed independently), and large-scale data processing frameworks like Spark that partition a dataset across a cluster. These predate deep learning and are the broader field of [distributed computing](https://howaiworks.ai/glossary/distributed-computing); AI is the workload that made massively parallel hardware mainstream.

## Key Concepts

- **Speedup and efficiency**: *Speedup* is sequential time divided by parallel time; *efficiency* is speedup divided by the number of processors. Efficiency near 1.0 means processors are well used; it falls as communication and Amdahl's sequential fraction take over.
- **Amdahl's law vs Gustafson's law**: Amdahl fixes the problem size and finds a speedup ceiling. Gustafson observes that in practice people *grow* the problem to fill available hardware — bigger models, bigger batches — so the parallel fraction rises with scale and the ceiling is less binding than Amdahl alone suggests. Both are true; they answer different questions.
- **Communication overhead**: The cost of moving data and synchronizing between devices. It grows with the number of devices and is the practical reason parallel scaling stops paying off, distinct from the theoretical Amdahl ceiling.
- **Memory bandwidth**: Matrix multiplication at scale is frequently limited not by arithmetic but by how fast weights can be fed from memory to the [GPU's](https://howaiworks.ai/glossary/gpu-computing) cores. A GPU can be arithmetically idle while waiting on memory — throughput without bandwidth to match is wasted.

## Challenges

The central difficulty is that the two ceilings above are real and cannot be engineered away, only managed. Amdahl's sequential fraction and communication overhead both mean that beyond some device count, adding hardware slows the *marginal* return to near zero — so the hard question in large-scale training is not "can we add more GPUs" but "where does the payoff stop," and it stops sooner than budgets assume.

**Load imbalance** is the quiet killer. Parallel work finishes only when the *slowest* participant finishes, so one device with slightly more work, or one pipeline stage that is heavier than the others, stalls everyone else. A pipeline whose stages are unevenly sized wastes GPUs on the bubble; a data-parallel job with uneven batch sizes waits on its straggler at every all-reduce.

**Debugging is genuinely harder** than sequential code because bugs depend on timing. Race conditions and deadlocks appear only under specific interleavings and often vanish when you add logging (which changes the timing), making them maddening to reproduce. A parallel program can produce correct output on ten runs and a silently wrong number on the eleventh.

**The memory wall** increasingly dominates. As arithmetic throughput has raced ahead, the bottleneck for large matmuls has shifted to memory bandwidth — feeding the cores, not the cores themselves. Techniques like FlashAttention exist precisely to restructure a computation so it moves less data between memory levels, trading extra arithmetic (which is cheap and parallel) for fewer memory trips (which are the real constraint).

## Code Example

This shows the SIMD/SIMT idea directly: the same operation applied to a whole array at once, with no Python loop over elements. On a GPU, the elements of the array are processed by different cores in parallel; the code you write is identical whether it runs on one core or thousands.

```python
import numpy as np

# Sequential thinking: one element at a time (what we are NOT doing)
def elementwise_square_loop(x):
    out = np.empty_like(x)
    for i in range(len(x)):        # each iteration waits for the last
        out[i] = x[i] * x[i]
    return out

# Data-parallel thinking: one operation over the whole array
def elementwise_square_vectorized(x):
    return x * x                    # every element handled independently

data = np.arange(1_000_000, dtype=np.float32)
assert np.allclose(elementwise_square_loop(data),
                   elementwise_square_vectorized(data))
```

The two functions compute the same result, but the second expresses the work as a single parallel operation instead of a sequence of dependent steps. Moving from the loop to the vectorized form is the mental shift that lets the same code run on a GPU:

```python
import torch

x = torch.arange(1_000_000, dtype=torch.float32, device="cuda")
y = x * x   # dispatched across the GPU's cores; the elements are independent

# Matrix multiplication is the same idea at scale: N^3 independent
# multiply-adds, all issued to the Tensor Cores at once.
a = torch.randn(8192, 8192, device="cuda")
b = torch.randn(8192, 8192, device="cuda")
c = a @ b   # ~1.1 trillion FLOPs, none of them waiting on another
```

The `@` operator hides the parallelism, but that one line is where a GPU spends most of a training run — and where every idea in this article, from SIMT to Amdahl's law to the three kinds of model parallelism, is really about making that single operation, repeated billions of times, run as close to the hardware's peak as the sequential parts and the memory wall allow.

## Frequently Asked Questions

### What is the difference between parallel processing and concurrency?

Concurrency is about structure: dealing with many tasks at once by interleaving them, even on a single core that only ever runs one at a time. Parallelism is about execution: literally running many computations in the same instant on separate hardware. A single-core laptop can be concurrent but never parallel; a GPU is massively parallel.

### Why do AI models need GPUs instead of CPUs?

Neural network training is dominated by matrix multiplication, which is millions of independent multiply-add operations with no dependencies between them. A CPU has tens of cores; an NVIDIA H100 has 16,896 CUDA cores and 528 Tensor Cores, so it can run thousands of those multiply-adds at once. The work is a perfect fit for the hardware.

### What is Amdahl's law and why does it matter for parallel processing?

Amdahl's law says the speedup from parallelization is capped by the fraction of work that must stay sequential. If 95% of a program parallelizes perfectly, the maximum speedup is 1 / (1 - 0.95) = 20x, no matter how many processors you add. The 5% that cannot be split becomes the whole cost.

### Why does one slow GPU stall an entire training job?

Because parallel work finishes only when the slowest participant finishes. A data-parallel job waits on its straggler at every all-reduce, and a pipeline whose stages are unevenly sized wastes every other device on the bubble. Load imbalance is a separate problem from Amdahl's ceiling: the sequential fraction limits what parallelism could ever buy, while imbalance wastes the parallelism you already paid for.

### Does adding more GPUs always make training faster?

No. Communication between devices grows with the number of them, and Amdahl's law caps the payoff of the parallel part. With 95% parallelizable work, going from 256 to 512 GPUs moves the speedup from about 18.6x to 19.3x while doubling the hardware bill. Past a point you buy communication overhead, not speed.

## Related

### Related terms

- [GPU Computing](https://howaiworks.ai/glossary/gpu-computing)
- [Distributed Training](https://howaiworks.ai/glossary/distributed-training)
- [Matrix Multiplication (GEMM)](https://howaiworks.ai/glossary/matrix-multiplication)
- [Distributed Computing](https://howaiworks.ai/glossary/distributed-computing)
- [Concurrency](https://howaiworks.ai/glossary/concurrency)
- [Vectorization](https://howaiworks.ai/glossary/vectorization)

---

Source: https://howaiworks.ai/glossary/parallel-processing — HowAIWorks.ai