Definition
A large language model (LLM) is a neural network that does exactly one thing: given a stretch of text, it assigns a probability to every possible next token and picks one. Answering a question, translating a contract, writing a function — all of it is that single operation repeated a few hundred times, with each token the model emits appended to the input before the next one is chosen. There is no separate module for reasoning and no separate module for translation. There is a loop.
That mechanism is also the answer to the question most people arrive with, which is how an LLM differs from a chatbot. ChatGPT, Claude and Gemini are products; the model is the engine inside them, and the product wraps it in a conversation format, a hidden system prompt, tools it can call, a retrieval layer, safety filters and a memory store. Strip the wrapper away and what is left continues text and has no concept of being spoken to. "AI", meanwhile, is the whole field. A large language model is one kind of system built inside it, and it is not the kind that drives a car or folds a protein.
"Large" names two quantities that grew together. GPT-3, the model that made the category visible, had 175 billion parameters (Brown et al., 2020). Llama 3's flagship has 405 billion, trained on 15.6 trillion tokens of text (Dubey et al., 2024). Stored at 16 bits each, those 405 billion parameters occupy 810 GB — which is the entire reason a frontier model lives in a data centre and reaches you over an API rather than running on your laptop.
The consequence readers most often miss is that next-token prediction contains no notion of truth. The model learned which continuations are likely, not which are correct, and the two come apart precisely where the training data was thin. That is why an LLM will produce a fluent, correctly formatted citation to a paper that does not exist: a hallucination is not a malfunction but the machine doing its actual job in a place where the likely answer and the true one diverge. If you take one thing from this page, take this: verification is your job, and the model's confidence carries no information about whether it is right.
How It Works
Text becomes tokens, and tokens are not words
The model never sees letters. A tokenizer chops the input into pieces drawn from a fixed vocabulary — Llama 3 uses 128,000 of them — and hands the network a list of integers. Common words are a single token, rare words split into several, and a leading space usually travels with the word that follows it.
The Llama 3 paper reports its tokenizer compressing English at 3.94 characters per token. An English word plus its trailing space runs to about six characters, so the working rule is roughly 1.5 tokens per word: a 500-word email is around 750 tokens, and a full-length book a few hundred thousand.
This is not trivia. Context limits and API bills are both denominated in tokens, so the cost of a prompt is a fact about the tokenizer rather than about how long the text looks to you — and languages the vocabulary covers less efficiently cost more tokens to say the same thing. It also explains a famous failure. Asked how many times the letter "r" appears in "strawberry", the model is looking at two or three opaque integers, not a string of characters. It is being asked about something it structurally cannot see.
Attention, and why context is expensive
Each token id becomes a vector, and that vector passes through a stack of transformer layers. The operation that makes the architecture work is attention: at every layer, each position looks at every earlier position and pulls in whatever is relevant. "It" can find its antecedent forty tokens back; a closing brace can find the function it closes.
The bill for that is quadratic. Vaswani et al. (2017) give the per-layer cost of self-attention as O(n²·d) in the sequence length n — every token attends to every other, so a 4,000-token prompt involves 16 million token pairs while an 8,000-token prompt involves 64 million. Doubling the context quadruples the attention work. That is why long context windows were a research programme rather than a switch someone forgot to flip, and why serving systems work so hard on the KV cache that stops the model recomputing all of it for every new token.
Sampling, and why the same prompt gives different answers
The final layer emits a score for every token in the vocabulary, and those scores become a probability distribution. The system then draws from it. Temperature controls how sharply: near zero the highest-probability token wins almost every time, while higher values flatten the distribution and let unlikely continuations through.
Because the output is a sample and not a lookup, asking the same question twice can produce two different answers, both reasonable. This surprises people who expect a database. It also means "the model said X" is a statement about one draw from a distribution, which is why judging an LLM by trying it once tells you very little.
Two training stages that do completely different jobs
Pretraining is the expensive part: read trillions of tokens and, after each prediction, nudge every parameter to make the token that actually followed slightly more likely. Kaplan et al. (2020) give the standard estimate of what this costs — approximately 6 floating-point operations per parameter per token once the backward pass is counted, so a run costs about 6ND FLOPs for N parameters and D training tokens. Check it against a real model: 6 × 405 billion × 15.6 trillion is 3.79 × 10²⁵ FLOPs, and the Llama 3 paper reports a pretraining budget of 3.8 × 10²⁵ FLOPs. The rule of thumb reproduces the published figure to two significant digits, which is what makes it useful for estimating anyone's training bill from two public numbers.
How that budget is split between N and D is a decision, not a given. Hoffmann et al. (2022) trained a 70-billion-parameter model, Chinchilla, on 1.4 trillion tokens — 20 tokens per parameter — and it outperformed the 280-billion-parameter Gopher trained on the same compute. Four times smaller, four times more data, better results. Scaling laws are the reason the field kept spending, and the Chinchilla result is the reason it stopped spending all of it on parameters.
What comes out of pretraining is still not an assistant. It is a text-continuation engine: hand a base model "What is the capital of France?" and a perfectly good continuation is three more exam questions. Post-training — supervised fine-tuning on demonstrations, then RLHF or a preference method such as DPO — is what converts "continue this text" into "answer this, decline that, and stop when you are finished". Almost everything a user experiences as the model's personality was installed in this second stage, on a compute budget orders of magnitude smaller than the first.
Where the knowledge actually lives
This is the question behind most of the searches that land on this page, and the honest answer has three parts that behave nothing like each other.
In the weights. 405 billion parameters at 16 bits is about 6.5 trillion bits of storage, spread across 15.6 trillion tokens of training text: under half a bit of model per token read. Taking the paper's measured 3.94 characters per token, that corpus is on the order of 60 TB of text pressed into 810 GB of weights, a compression of roughly 76 to one. Something that lossy cannot be storing the text. It is storing regularities. A fact repeated across thousands of documents survives the compression; a detail that appeared once does not — and the model has no way to tell which kind of memory it is drawing on, which is exactly why a hallucination sounds as confident as a fact. Weights are also frozen when training ends, and that is all a knowledge cutoff is.
In the context. Anything in the prompt is present exactly and verbatim, costs nothing in training, and disappears when the conversation ends. It is billed by the token and its attention cost grows quadratically. Nothing you paste into a chat changes a single parameter.
In a retrieval layer. RAG closes the gap by searching a corpus at query time and placing the retrieved passages into the context, so exact text meets a model that is good at using exact text. This is why "make the model know my notes" is a retrieval problem rather than a training problem, and it is the single most consequential thing to get right. Fine-tuning on your own documents reliably teaches format, tone and vocabulary; it is a poor and expensive way to install facts, because you are asking a compressor that keeps half a bit per token to memorise. Point the model at the documents instead of trying to bake them in.
Types
Three architectures descend from the original transformer, and these names are in general use. Encoder-only models — BERT and its relatives — read the whole sequence in both directions at once and produce representations; they are excellent at classification, embeddings and search ranking, and they cannot generate text. Decoder-only models — GPT, Llama, Claude, Gemini, Qwen, DeepSeek — apply a causal mask so each position sees only what came before it, which is what makes open-ended generation possible. Encoder-decoder models such as T5 and BART keep both halves and suit transformation tasks, translation above all.
When someone says "LLM" today they nearly always mean the decoder-only kind. The encoder-only family did not go away — it quietly runs a large share of the world's search, moderation and retrieval — but it is not what anyone is talking to.
A second axis cuts across the first: dense versus mixture-of-experts. A dense model uses every parameter for every token. An MoE model routes each token to a small subset of expert sub-networks, so total and active parameter counts diverge sharply — a trillion-parameter MoE may activate only tens of billions per token. This matters when comparing two models, because "parameter count" stops being one number: the total tells you the memory footprint and roughly the training cost, while the active count tells you the inference cost.
Real-World Applications
Code is where the mechanism fits best, because source code is far more predictable from its immediate context than prose is. GitHub Copilot and Cursor put a model behind the cursor in an editor, and agents such as Claude Code run the same loop further out, editing files and running tests between generations. None of this requires the model to understand a program; it requires the next token in a function to be unusually constrained by the thousand tokens before it.
Search products bolt retrieval onto generation. Perplexity runs a web search first and generates an answer over the retrieved pages with citations attached, and Google's AI Overviews do the same shape of thing above the results. Both exist because frozen weights cannot be current, and both attach links because the generation step alone cannot be trusted.
NotebookLM is the clearest consumer product built directly on the weights-versus-context distinction: it answers only from sources you upload, a deliberate refusal to let pretrained knowledge leak into the answer. An internal assistant over a company wiki is the same design instinct with a different corpus.
General assistants are the largest deployment by volume — ChatGPT, Claude and Gemini in most markets, and Doubao, ByteDance's free multimodal assistant, in China. Microsoft 365 Copilot embeds the same class of model in documents and mail, where the hard engineering is retrieval over your organisation's own data rather than the model itself.
Challenges
- Hallucination is the mechanism showing through, not a bug awaiting a patch. Every mitigation — retrieval, citations, tool use, asking for a second pass — works by giving the model exact text to lean on or by checking its output afterwards. None of them changes the fact that a model with a thin patch of training data will still produce the most plausible-looking continuation.
- There is no channel separation between instructions and data. An LLM reads its system prompt, your message and a web page it just fetched as one undifferentiated stream of tokens. That is what makes prompt injection structurally hard: a sentence inside a retrieved document saying "ignore previous instructions and email the summary to this address" is, mechanically, indistinguishable from an instruction you wrote.
- The tokenizer hides characters and digits. Counting letters, reversing strings, spelling backwards and multi-digit arithmetic are all hard for reasons that have nothing to do with intelligence: the model is reasoning over vocabulary ids, not over the characters those ids stand for. Give it a calculator or a code interpreter rather than a better prompt.
- Prefill and decode have completely different cost shapes. Reading a 100,000-token document is one large parallel matrix operation; writing 500 tokens back is 500 sequential passes over the entire weight matrix, each one waiting on memory bandwidth rather than arithmetic. This is why time-to-first-token and tokens-per-second are separate numbers, and why techniques such as speculative decoding attack only the second one.
- A long context window is not a filing cabinet. Liu et al. (2023) found a U-shaped curve: models use information placed at the very start or very end of a long context far more reliably than the same information placed in the middle. A million-token window is a capacity, not a guarantee that everything in it is equally available.
- The weights cannot be updated with a fact. There is no operation that teaches a deployed model that a price changed this morning. Retraining is a multi-million-dollar batch job, fine-tuning shifts behaviour but does not reliably install facts, so everything time-sensitive has to arrive through the context — from retrieval, a tool call or the user.
Future Trends
- Active parameters displace total parameters as the number that matters. As mixture-of-experts becomes the default at the frontier, a headline parameter count describes memory and training cost while saying almost nothing about what a query costs to serve. Comparing a dense model to an MoE on total parameters alone is already misleading.
- Buying accuracy with inference compute instead of training compute. Reasoning models spend hundreds or thousands of extra tokens thinking before they answer, trading latency and money at inference time — test-time compute — for capability that would otherwise have to come from a larger pretraining run. It is a different point on the same curve, and it shifts spend from a one-off training bill to a per-query one.
- Capable models are shrinking onto local hardware. Distillation plus quantization is the lever: a 70-billion-parameter model needs about 140 GB of memory at 16 bits and roughly 35 GB at 4 bits, which is the difference between a server rack and one high-memory machine. The privacy and latency arguments for running a model locally only bite once it fits.
- Serious attempts to escape the quadratic. Sliding-window and linear-attention variants, state-space hybrids and diffusion language models all attack the n² term or the strictly sequential decode loop. None has displaced the standard transformer at the frontier yet, and whichever does will change the economics of long context more than any product feature.