AIHow AI Works

Pre-trained Models

A neural network someone else already trained on a huge general dataset, which you download and adapt to your task instead of training from scratch.

Definition

A pre-trained model is a neural network that someone else has already trained on a huge, general dataset, which you download and adapt to your own task instead of training one from scratch. The point is economic: pre-training a large model costs enormous compute and money exactly once, and adapting it afterwards costs a tiny fraction of that, so almost nobody starts from random weights anymore when a suitable pre-trained model already exists.

(adsbygoogle = window.adsbygoogle || []).push({});

Concretely, when you fetch BERT or a ResNet off a model hub, you are not getting an empty network — you are getting one whose weights already encode grammar, or edges and textures and object shapes, learned from billions of examples. Your job shrinks from “teach a network language from nothing” to “nudge a network that already understands language toward my specific problem.” That shift is the reason applied AI became affordable for teams that could never afford to pre-train.

Two neighbouring terms are easy to confuse with this one. A foundation model is a particular kind of pre-trained model — one large and general enough to serve many downstream tasks; every foundation model is pre-trained, but a small vision model trained only to classify ImageNet photos is pre-trained without being a foundation model. Transfer learning is the general principle of reusing knowledge from one task on another, and fine-tuning is the most common method of doing that reuse. A pre-trained model is the reusable artifact those two ideas act on.

(adsbygoogle = window.adsbygoogle || []).push({});

How It Works

The lifecycle has two stages that differ by orders of magnitude in cost. The first stage, pre-training, teaches a network general structure from a massive corpus, usually with self-supervised learning so no human has to label the data — the model predicts held-back words or image patches and learns representations as a side effect. The second stage, adaptation, takes those frozen or nearly-frozen weights and specializes them on a much smaller, task-specific dataset.

Why is the first stage so expensive? A useful rule of thumb is that training compute scales as roughly 6 × N × D floating-point operations, where N is the parameter count and D is the number of training tokens. Both are enormous for a modern model — hundreds of billions of parameters trained on tens of trillions of tokens — so the product lands around 10²⁵ to 10²⁶ operations, the kind of workload that occupies thousands of accelerators for weeks. For a concrete anchor: Meta reports that pre-training Llama 3.1 405B on 15.6 trillion tokens took 3.8 × 10²⁵ FLOP (2024), and Epoch AI’s independent tracking counts more than 30 models trained above the 10²⁵ FLOP mark, the largest around 5 × 10²⁶. That is a bill you pay once and then amortize across every downstream user of the weights.

Adaptation is cheap by comparison because it touches far less. In the extreme, parameter-efficient fine-tuning (PEFT) freezes the original weights entirely and trains a small number of new ones. The LoRA method (Hu et al., 2021) is the canonical example: instead of updating a weight matrix directly, it learns a low-rank pair of small matrices that adjust it. The paper reports that compared to fine-tuning GPT-3 175B with Adam, LoRA “can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times.” In their results table, full fine-tuning updates all 175,255.8M parameters of GPT-3, while LoRA reaches matching quality by training as few as 4.7M — about 0.003% of the model — or 37.7M in a larger configuration. That is the whole argument for pre-training in one number: a change of roughly 37,000× in what you have to train.

The reason this works at all is that pre-training front-loads the hard, general learning. The features a network needs — how words relate, what an edge or a face looks like — are shared across almost every task in a modality, so they only have to be learned once. Fine-tuning then re-weights and slightly reshapes those features rather than discovering them, which is why it converges on a small dataset that would be nowhere near enough to train the same network from scratch.

Real-World Applications

The dominant real-world use of pre-trained models is simply the default workflow of modern applied machine learning: download, then adapt. A team building a medical-imaging classifier does not gather millions of labeled scans and train a vision network from zero — they take a model pre-trained on ImageNet, whose early layers already detect edges and textures, and fine-tune it on a few thousand annotated scans. The pre-trained edges transfer directly; only the final task-specific layers need substantial retraining.

In language, the same pattern runs on Hugging Face’s model hub, which hosts millions of pre-trained checkpoints that anyone can fetch — more than two million as of 2026. A company classifying support tickets or extracting fields from contracts starts from a pre-trained encoder like BERT — 110M parameters in its base configuration, 340M in its large one, per the original paper — and fine-tunes it on their in-house labels. Starting from those weights routinely reaches usable accuracy on a dataset a thousand times smaller than pre-training used.

Parameter-efficient adaptation has made this even more granular. Because a LoRA adapter is tiny, practitioners keep one frozen base model in memory and swap small task-specific adapters in and out — the community around image models like Stable Diffusion, for instance, shares thousands of LoRA adapters that each teach a shared base model a particular style or subject without anyone re-training the base. One expensive pre-trained artifact, many cheap specializations, is the whole economy of the ecosystem.

Challenges

The failure mode that catches beginners is treating fine-tuning as free and safe. It is neither, and the two things that break both come from adapting on too little data.

Catastrophic forgetting is the first. If you fine-tune every parameter of a pre-trained model hard on a narrow dataset, gradient updates can overwrite the very general knowledge that made the model worth starting from — the network gets better at your 500 examples and measurably worse at everything else. This is exactly why PEFT methods and layer freezing exist: keeping most weights fixed protects the pre-trained knowledge from being trampled by a small task signal. The related term catastrophic forgetting covers the mechanism in depth.

Overfitting is the second and closely linked. A model with hundreds of millions of parameters can simply memorize a few thousand fine-tuning examples, scoring near-perfect on your training set and failing on anything new. A network trained from scratch on so little data would obviously overfit; the danger with pre-trained models is that strong starting weights mask the problem — training loss looks healthy while the model quietly loses generality.

Beyond training dynamics, pre-trained models carry their origins with them. Whatever bias sat in the pre-training corpus ships inside the weights and can survive fine-tuning, so a downstream user inherits problems they never introduced. Licensing is a practical trap of the same kind — “open-weight” does not always mean free for commercial use, and the terms attach to the checkpoint you downloaded, not to your fine-tuned copy. And a pre-trained model is a snapshot of the world at training time: it knows nothing that happened after its data cutoff unless you add retrieval or continue training.

The clearest trend is that parameter-efficient adaptation is becoming the default, not the exception. As base models grow, full fine-tuning of every weight becomes impractical for most teams on cost and memory grounds, while methods like LoRA and its quantized variant QLoRA push the trainable fraction well under 1% and let a single consumer GPU adapt a model that would take a cluster to train. The economic gap this page describes — expensive pre-training, cheap adaptation — is widening, which pushes ever more of the ecosystem toward reusing a small number of heavily pre-trained bases rather than training new ones.

Common questions

What is the difference between a pre-trained model and a foundation model?

A foundation model is a large pre-trained model that is general enough to serve many different tasks, so every foundation model is pre-trained but not every pre-trained model is a foundation model. A ResNet trained only to classify ImageNet photos is pre-trained; it is not a foundation model because it does one narrow thing.

Why not just train a model from scratch?

Pre-training a large model is a one-time megaproject: cost scales with parameters multiplied by training tokens, so a modern model costs enormous compute to train once. Adapting an already-trained model to your task reuses all of that work, which is why almost no one trains from scratch when a suitable pre-trained model exists.

How much of a pre-trained model does fine-tuning actually change?

It depends on the method. Full fine-tuning updates every parameter, but parameter-efficient methods like LoRA train a tiny fraction. On GPT-3 175B, the LoRA paper reports training as few as 4.7M parameters instead of all 175 billion, roughly 0.003%, while matching full fine-tuning quality.

Can fine-tuning damage a pre-trained model?

Yes. Fine-tuning on a small dataset can cause catastrophic forgetting, where the model overwrites the general knowledge it learned in pre-training, and overfitting, where it memorizes your few examples instead of learning the task. Freezing most of the network or using parameter-efficient methods reduces both risks.

Where do pre-trained models come from?

You download them. Public hubs like Hugging Face host millions of pre-trained checkpoints — more than two million as of 2026 — and labs release open-weight models you can fetch and adapt. You start from those weights rather than from random initialization.