Weights

Definition

A weight is a single learned number that sets the strength of one connection between two neurons. A neuron takes each of its inputs, multiplies it by that input's weight, adds up the results, adds one more number called the bias, and passes the total through an activation function — so the weights are the dials that decide how much each input matters. A network can have billions of these dials, and "learning" is nothing more than adjusting them, a little at a time, so the network's outputs get closer to the right answers.

The consequence worth carrying away first: the weights are the model. Everything a trained network knows lives in these numbers and nowhere else. When a lab publishes an "open-weights" model like Llama or DeepSeek, the release is literally a file of these numbers — download the weights and you have the model; delete them and the architecture is an empty shell that does nothing. Training is the process of finding good values for the weights; inference is the process of putting numbers through the weights you already found.

How It Works

Start with a single neuron. Give it three inputs x = [0.5, -0.2, 0.1], three weights w = [0.9, 0.4, -0.7], and a bias b = 0.15. The neuron computes a weighted sum: it multiplies each input by its weight and adds the products, (0.9 × 0.5) + (0.4 × -0.2) + (-0.7 × 0.1) = 0.45 + (-0.08) + (-0.07) = 0.30, then adds the bias to get 0.45. In symbols this is y = Σ wᵢxᵢ + b — every neuron in every network, from a 1980s perceptron to a trillion-parameter transformer, is doing this one arithmetic operation, just billions of times. A large positive weight says "this input matters a lot, and in the same direction as the output"; a weight near zero says "ignore this input"; a negative weight says "this input pushes the output the other way."

A whole layer is just many neurons wired to the same inputs, so its weights form a grid. A dense layer connecting 784 inputs (the pixels of a 28×28 image) to 128 neurons needs one weight for every input-to-neuron pair: 784 × 128 = 100,352 weights, plus one bias per neuron for 128 more, giving 100,480 learned numbers in a single small layer. Stack a few dozen layers and the count runs into the billions — which is exactly where names like "7B" come from. Weights are by far the largest share of a model's parameters; the biases, one per neuron, are a rounding error next to them.

How the values are found

The weights do not start meaningful. Before training they are set to small random numbers, and the network's outputs are noise. Training then repeats a loop: run inputs forward through the current weights to get a prediction, measure how wrong it is with a loss function, use backpropagation to work out which direction each weight should move to make the loss smaller, and nudge every weight a tiny step that way with gradient descent. One step barely changes anything; millions of steps over billions of examples slowly turn the random numbers into a network that recognizes faces or writes code. Nothing about the architecture changes during this process — only the numbers in the weight grids.

Weights are not the other numbers in a network

Three things get confused with weights, and keeping them apart is most of understanding the term:

• Weights vs. biases. A weight multiplies an input, scaling how much it counts. A bias is added once, after all the multiplications, and shifts the neuron's output up or down no matter what the inputs are. Both are learned, but there is one bias per neuron and one weight per connection, so weights vastly outnumber biases.
• Weights vs. activations. Weights are fixed once training ends. Activations are the changing values that flow through the network on each input — the output of the weighted sum, different for every image or sentence you feed in. The weights are the pipe; the activations are the water. When people talk about a model "thinking," they mean activations moving through frozen weights.
• Weights vs. hyperparameters. The learning rate, the number of layers, the batch size — those are hyperparameters, chosen by the engineer before training and never touched by gradient descent. Weights are learned by the machine; hyperparameters are set by you.

Why storing them is the whole ballgame

Because a model is its weights, deploying it is mostly a question of where those numbers fit. Each weight is stored at some numeric precision. In FP32 (single precision) each weight takes 4 bytes; in FP16 or BF16 (half precision) each takes 2 bytes. A 7-billion-parameter model is therefore about 7 × 10⁹ × 4 = 28 GB in FP32 and about 14 GB in FP16 — the same numbers, at half the bit-width, taking half the memory. That halving is why quantization exists and why model size is quoted in gigabytes as often as in parameter counts: cut the bytes per weight and a model that needed two GPUs suddenly fits on one, at some cost to precision. The weights did not change; how many bits you spend describing each one did.

Real-World Applications

• Open-weights model releases. When Meta ships Llama 3.1 or DeepSeek ships DeepSeek-V3, the deliverable is a set of weight files (usually in the .safetensors format) plus a few kilobytes of code. "Open weights" means those numbers are downloadable, so anyone can run, fine-tune, or inspect the model on their own hardware — the license terms are a separate matter from the weights being published. This is the most literal proof that the weights are the model: the entire released artifact is a directory of them.
• Quantized deployment. Serving a model on a phone or a single consumer GPU almost always means shipping the same trained weights at lower precision. Storing a 7B model's weights at 4 bits instead of FP16's 16 halves memory twice over — from ~14 GB toward ~3.5 GB — so it fits where the full-precision weights never could. The weights carry the knowledge; the precision is a deployment choice made per device.
• Fine-tuning and transfer learning. Rather than initialize weights randomly and train from scratch (millions of dollars of compute), teams start from a published model's weights and adjust them on a smaller, task-specific dataset. The pretrained weights already encode general language or vision; fine-tuning moves them a short distance toward a specialty. Techniques like LoRA go further, freezing the original weights entirely and learning a small set of new ones alongside them.

Challenges

Bad initialization breaks the network before it starts. The tempting idea of setting every weight to zero is a classic trap. If all weights in a layer are identical, every neuron in that layer computes the same weighted sum, receives the same gradient during backpropagation, and updates by the same amount — so they remain identical forever. A layer of 128 neurons that can only ever represent one neuron is useless, and no amount of training fixes it because the symmetry never breaks. Random initialization exists precisely to give each neuron a different starting point so they can specialize. This is not an exotic edge case; it is the first thing initialization schemes like Xavier and Kaiming are designed to avoid.

Weights can explode or vanish as they compound. A deep network multiplies its inputs by weights layer after layer. If the weights are consistently a bit larger than the network can absorb, the values grow with each layer and the numbers blow up to infinity (exploding); if they are consistently a bit small, the signal shrinks toward zero and the early layers stop learning (vanishing). Both show up as training that either diverges into NaN or flatlines with no progress, and both are reasons initialization is tuned to the size of each layer rather than picked arbitrarily.

The weights are opaque. A single weight of 0.42 between two neurons in layer 30 means nothing a human can read. Knowledge in a network is spread across billions of weights with no one number holding an interpretable fact, which is why interpretability is hard and why you cannot simply "edit the weight for a wrong belief." It is also why the weights are large: there is no shorter description of what the model learned than the numbers themselves.

Code Example

A neuron is one weighted sum, a layer is a grid of them, and all-zero initialization is a trap you can watch fail. Running the block as published prints:

# One neuron: multiply each input by its weight, sum, add the bias.
inputs  = [0.5, -0.2, 0.1]
weights = [0.9,  0.4, -0.7]   # the three learned numbers
bias    = 0.15                # one more learned number

z = sum(w * x for w, x in zip(weights, inputs)) + bias
print("weighted sum z =", round(z, 3))

# A dense layer wiring 784 inputs to 128 neurons is just 784x128 of these
# weights, plus one bias per neuron.
n_in, n_out = 784, 128
print("layer weights:", n_in * n_out, "+ biases:", n_out)

# Why you never initialise all weights to zero: every neuron then computes the
# same output and gets the same gradient, so they stay identical forever --
# 128 neurons that can never learn 128 different things.
layer = [[0.0] * n_in for _ in range(n_out)]        # all-zero init
outs = [sum(w * x for w, x in zip(row, [1.0] * n_in)) for row in layer[:4]]
print("all-zero init, first 4 neuron outputs:", outs)

Output:

weighted sum z = 0.45
layer weights: 100352 + biases: 128
all-zero init, first 4 neuron outputs: [0.0, 0.0, 0.0, 0.0]

The four identical zeros are the symmetry problem in miniature: with no variation in the starting weights, there is nothing to tell the neurons apart, so training cannot pull them in different directions.