Tensor Operations

Definition

A tensor is just an n-dimensional array of numbers: a single number (a scalar) is a 0-D tensor, a list of numbers (a vector) is 1-D, a grid (a matrix) is 2-D, and a stack of grids is 3-D or higher. Tensor operations are the handful of array manipulations — elementwise math, matrix multiplication, reshaping, broadcasting, and reduction — that every neural network is built out of. That is the whole idea in one line: tensors are the data structure of deep learning, and the operations on them are what a network actually computes.

When people say a model "processes an image" or "reads a sentence", underneath that description the image is a tensor of pixel values and the sentence is a tensor of numbers, and every layer of the neural network turns one tensor into the next by chaining these few operations. There is no separate magic step. Learn what the five operation families do to the shape of a tensor and you can read the architecture of almost any model, because a model is a pipeline of shape transformations with numbers flowing through it.

A tensor's shape is the tuple of its dimension sizes, and it is the single most important thing to track. A batch of 32 greyscale images that are 28×28 pixels each is a tensor of shape (32, 28, 28); flatten each image and it becomes (32, 784), because 28 × 28 = 784. The 32 is the batch dimension — how many independent examples you push through at once — and everything a layer does, it does to all 32 in parallel. Keeping shapes straight in your head is most of what it takes to reason about a network, and losing track of them is where most deep-learning bugs come from.

How It Works

Almost every layer in a neural network is one core operation: multiply the incoming data by a matrix of learned weights, add a bias, and apply a nonlinearity. Follow the shapes through a single linear layer and the whole mechanism is visible. Take the batch of 32 flattened images, shape (32, 784), and a weight matrix of shape (784, 128) — 784 numbers coming in, 128 going out. Matrix-multiply them and you get an output of shape (32, 128): each of the 32 examples has been turned from a 784-number vector into a 128-number vector. The inner dimensions (784 and 784) have to match and they cancel; the outer dimensions (32 and 128) survive to become the output shape. Then a bias vector of shape (128,) is added to every row.

That bias-add is broadcasting in action. The activation is (32, 128) and the bias is (128,) — different shapes — but the operation still works because broadcasting aligns shapes from the right and stretches the missing left-hand dimension: the length-128 bias is applied identically to all 32 rows, as if it had been copied into a (32, 128) tensor, but without actually spending the memory to copy it. The same rule lets you add a scalar to a whole tensor or a per-channel scale to an image. Broadcasting is what makes array code short; it is also, as the Challenges section shows, what makes some shape bugs invisible.

The cost of that matrix multiplication is worth knowing because it dominates the compute in most networks. Multiplying an M × K matrix by a K × N matrix takes roughly 2 · M · N · K floating-point operations — the factor of two because each output element is a sum of K products, and each term is one multiply plus one add. For our layer that is 2 × 32 × 128 × 784 = 6,422,528 FLOPs for a single tiny layer on a single batch. A real transformer has hundreds of far larger matrix multiplies per token, which is why the FLOP count of a forward pass climbs into the billions and why the hardware chapter of deep learning is really a chapter about doing matrix multiplication fast.

Two more operations round out the toolkit. A reduction collapses a dimension by aggregating along it — a mean(axis=1) over our (32, 128) activations produces a (32,) tensor, one number per example — and reductions are how losses, norms, and pooling layers turn many numbers into few. A reshape relabels the same block of memory with new dimensions without moving any data, so turning (32, 784) back into (32, 28, 28) is essentially free. That is different from a transpose, which reorders which axis is which and therefore breaks the memory order; the transposed tensor is non-contiguous, and an operation that needs contiguous memory has to copy it first. Reshape is a re-label; transpose is a re-arrangement, and confusing the two is a common source of silent slowdowns.

Types

The dozens of functions in a tensor library fall into a few families, grouped by what they do to a tensor's shape. That grouping is the useful mental model, not a list of names to memorise:

• Elementwise operations keep the shape identical and apply one function to every element independently: adding two tensors, multiplying by a scalar, or running an activation function like ReLU. Because every element is independent, these are the most trivially parallel operations there are.
• Matrix multiplication (and its batched and convolutional cousins) is the one operation that mixes information across elements — every output number is a weighted sum of many inputs. It is where the learned weights live, and it is the expensive one.
• Reductions shrink a tensor by aggregating along one or more axes: sum, mean, max, and the norms and softmaxes built from them. They turn a dimension into a single summary number.
• Shape operations move no arithmetic at all: reshape, transpose, slice, concatenate, and broadcast expansion just rearrange or re-view existing numbers so the next real operation lines up.

A network is mostly matrix multiplications for the heavy lifting, elementwise nonlinearities between them, reductions to pool and to compute the loss, and shape operations as the plumbing that connects everything.

Real-World Applications

Every mainstream deep-learning framework is, at bottom, a tensor-operation engine, and the frameworks people actually build on make this explicit. PyTorch's central object is torch.Tensor and NumPy's is ndarray; both expose exactly the elementwise, matmul, reduction, and reshape operations above, and both record the operations so gradients can flow back through them during backpropagation. Writing a model in these libraries is writing a sequence of tensor operations.

Because matrix multiplication and elementwise math are so parallel, they are the reason specialised hardware exists. A GPU devotes thousands of small cores to running the same arithmetic on many array elements at once — precisely the parallel-processing shape of these operations — which is why training that would take weeks on a CPU finishes in hours. Google's Tensor Processing Unit goes further, building a hardware systolic array whose entire job is matrix multiplication; the chip is named after the operation it accelerates. When a data-centre operator talks about buying compute for AI, the unit they are really buying is tensor-operation throughput.

The operations are also what a practitioner debugs and profiles day to day. When a training run runs out of GPU memory, the culprit is the size of the intermediate tensors an operation produces; when it runs slowly, a profiler points at which matrix multiplication or which non-contiguous transpose is the bottleneck. Understanding the operations is not academic — it is what lets an engineer read a stack trace, fix a shape, and cut a model's memory or latency.

Challenges

The single most common bug in deep learning is a shape mismatch: two tensors whose dimensions do not line up for the operation you asked for. Ask to add a (32, 128) tensor and a length-64 vector and the library raises an error immediately — operands could not be broadcast together with shapes (32,128) (64,) — and stops. This is the good failure, because it is loud and it points at the exact line.

The dangerous failure is a silent wrong-broadcast: shapes that happen to be compatible under the broadcasting rules but are not the ones you meant. Suppose you intend a per-example bias of shape (32,) but accidentally hand the operation a (32, 1) column and a (128,) row. Broadcasting aligns from the right, stretches both size-1 and missing dimensions, and produces a perfectly valid (32, 128) result — no error, no warning, and a tensor that is 128 wide where you expected 32. The program runs to completion and computes nonsense; the loss looks a little off, the model trains a little worse, and nothing tells you why. Silent broadcasts that produce the wrong shape are one of the hardest classes of deep-learning bug precisely because the tooling stays quiet. The defence is to assert the shape you expect after any operation you are unsure of, and to prefer explicit named dimensions over relying on the broadcast rules to guess your intent.

The other recurring challenge is memory and layout. Intermediate tensors — the activations a network holds so it can compute gradients — dwarf the model's weights during training, which is why batch size is capped by GPU memory rather than by anything mathematical. And the reshape-versus-transpose distinction bites here too: a transpose that leaves a tensor non-contiguous forces a hidden copy before the next matrix multiply, quietly doubling memory traffic. Neither of these is exotic; both are the everyday tax of thinking in tensors, and both are why memory profiling is a routine part of building models.

Code Example

This walks a batch through a single linear layer with NumPy, printing the shape after each operation so you can watch the transformations. Every number in the output is real — the block was run exactly as shown.

import numpy as np

# A batch of 32 images, each flattened to a 784-vector (28x28 pixels),
# fed through one linear layer with 128 output units.
x = np.random.randn(32, 784)   # (batch, in_features)
W = np.random.randn(784, 128)  # (in_features, out_features)
b = np.random.randn(128)       # one bias per output unit

h = x @ W + b                  # matmul, then broadcast-add the bias
print("x", x.shape, "@ W", W.shape, "->", (x @ W).shape)
print("+ b", b.shape, "->", h.shape)   # (128,) aligns to the last axis

# FLOP cost of the matmul: 2 * M * N * K (a multiply and an add per term)
M, K = x.shape
_, N = W.shape
print("matmul FLOPs = 2*M*N*K =", 2 * M * N * K)

# Reshape just relabels the same buffer; transpose reorders it.
img = x.reshape(32, 28, 28)
print("reshape same data ->", img.shape,
      "contiguous:", img.flags["C_CONTIGUOUS"])
print("transpose ->", img.transpose(0, 2, 1).shape,
      "contiguous:", img.transpose(0, 2, 1).flags["C_CONTIGUOUS"])

# Reduction collapses an axis: mean over features -> one number per row.
print("h.mean(axis=1) ->", h.mean(axis=1).shape)

Output:

x (32, 784) @ W (784, 128) -> (32, 128)
+ b (128,) -> (32, 128)
matmul FLOPs = 2*M*N*K = 6422528
reshape same data -> (32, 28, 28) contiguous: True
transpose -> (32, 28, 28) contiguous: False
h.mean(axis=1) -> (32,)

Read the output top to bottom and every claim above is visible: the matmul cancels the shared 784 and leaves (32, 128); the length-128 bias broadcasts across all 32 rows; the same layer costs 6,422,528 FLOPs; the reshape keeps the data contiguous while the transpose does not; and the reduction collapses the 128 features to a single number per example. That is deep learning in miniature — data flowing through a chain of shape transformations — and a full model is this pattern repeated hundreds of times.