Term tag: MoE

Mixture-of-Experts (MoE)

The architecture behind "671B total, 37B active": hundreds of expert sub-networks, a router that picks a few per token, and a memory bill that never shrinks.

Model Size

Model size is a model's parameter count. What that one number predicts — memory, cost per token, capability — worked on real models from 110M to trillions.

Parameters

The learned numbers inside a model — what '70 billion parameters' actually means, and how the count converts directly into the memory a GPU must hold.