Term tag: parameters

Layers in Neural Networks

A layer is one weight matrix plus a nonlinearity. How it differs from a neuron and a parameter, why stacking helps, and how many layers real models have.

Model Size

Model size is a model's parameter count. What that one number predicts — memory, cost per token, capability — worked on real models from 110M to trillions.

Parameters

The learned numbers inside a model — what '70 billion parameters' actually means, and how the count converts directly into the memory a GPU must hold.

Weights

A weight is a single learned number setting the strength of a connection between two neurons; together, the weights are the trained model.