---
source: 'https://howaiworks.ai/glossary/pre-trained-models'
section: glossary
title: Pre-trained Models
description: >-
  A neural network someone else already trained on a huge general dataset, which
  you download and adapt to your task instead of training from scratch.
tags:
  - pre-trained models
  - transfer learning
  - Fine-tuning
  - Foundation Models
  - LoRA
  - Machine Learning
category: Machine Learning
datePublished: '2025-07-16'
lastUpdated: '2026-07-24'
---

# Pre-trained Models

> A neural network someone else already trained on a huge general dataset, which you download and adapt to your task instead of training from scratch.

## Definition

A pre-trained model is a [neural network](https://howaiworks.ai/glossary/neural-network) that someone else has already trained on a huge, general dataset, which you download and adapt to your own task instead of training one from scratch. The point is economic: pre-training a large model costs enormous compute and money exactly once, and adapting it afterwards costs a tiny fraction of that, so almost nobody starts from random weights anymore when a suitable pre-trained model already exists.

Concretely, when you fetch [BERT](https://howaiworks.ai/glossary/transformer) or a ResNet off a model hub, you are not getting an empty network — you are getting one whose weights already encode grammar, or edges and textures and object shapes, learned from billions of examples. Your job shrinks from "teach a network language from nothing" to "nudge a network that already understands language toward my specific problem." That shift is the reason applied AI became affordable for teams that could never afford to pre-train.

Two neighbouring terms are easy to confuse with this one. A [foundation model](https://howaiworks.ai/glossary/foundation-models) is a *particular kind* of pre-trained model — one large and general enough to serve many downstream tasks; every foundation model is pre-trained, but a small vision model trained only to classify ImageNet photos is pre-trained without being a foundation model. [Transfer learning](https://howaiworks.ai/glossary/transfer-learning) is the general principle of reusing knowledge from one task on another, and [fine-tuning](https://howaiworks.ai/glossary/fine-tuning) is the most common *method* of doing that reuse. A pre-trained model is the reusable artifact those two ideas act on.

## How It Works

The lifecycle has two stages that differ by orders of magnitude in cost. The first stage, **pre-training**, teaches a network general structure from a massive corpus, usually with [self-supervised learning](https://howaiworks.ai/glossary/self-supervised-learning) so no human has to label the data — the model predicts held-back words or image patches and learns representations as a side effect. The second stage, **adaptation**, takes those frozen or nearly-frozen weights and specializes them on a much smaller, task-specific dataset.

Why is the first stage so expensive? A useful rule of thumb is that training compute scales as roughly `6 × N × D` floating-point operations, where `N` is the parameter count and `D` is the number of training tokens. Both are enormous for a modern model — hundreds of billions of parameters trained on tens of trillions of tokens — so the product lands around 10²⁵ to 10²⁶ operations, the kind of workload that occupies thousands of accelerators for weeks. For a concrete anchor: Meta reports that pre-training Llama 3.1 405B on 15.6 trillion tokens took 3.8 × 10²⁵ FLOP (2024), and Epoch AI's independent tracking counts more than 30 models trained above the 10²⁵ FLOP mark, the largest around 5 × 10²⁶. That is a bill you pay once and then amortize across every downstream user of the weights.

Adaptation is cheap by comparison because it touches far less. In the extreme, **[parameter-efficient fine-tuning (PEFT)](https://howaiworks.ai/glossary/lora)** freezes the original weights entirely and trains a small number of new ones. The LoRA method (Hu et al., 2021) is the canonical example: instead of updating a weight matrix directly, it learns a low-rank pair of small matrices that adjust it. The paper reports that compared to fine-tuning GPT-3 175B with Adam, LoRA "can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times." In their results table, full fine-tuning updates all 175,255.8M parameters of GPT-3, while LoRA reaches matching quality by training as few as **4.7M** — about **0.003%** of the model — or 37.7M in a larger configuration. That is the whole argument for pre-training in one number: a change of roughly 37,000× in what you have to train.

The reason this works at all is that pre-training front-loads the hard, general learning. The features a network needs — how words relate, what an edge or a face looks like — are shared across almost every task in a modality, so they only have to be learned once. Fine-tuning then re-weights and slightly reshapes those features rather than discovering them, which is why it converges on a small dataset that would be nowhere near enough to train the same network from scratch.

## Real-World Applications

The dominant real-world use of pre-trained models is simply the default workflow of modern applied machine learning: **download, then adapt.** A team building a medical-imaging classifier does not gather millions of labeled scans and train a vision network from zero — they take a model pre-trained on [ImageNet](https://howaiworks.ai/glossary/computer-vision), whose early layers already detect edges and textures, and fine-tune it on a few thousand annotated scans. The pre-trained edges transfer directly; only the final task-specific layers need substantial retraining.

In language, the same pattern runs on Hugging Face's model hub, which hosts millions of pre-trained checkpoints that anyone can fetch — more than two million as of 2026. A company classifying support tickets or extracting fields from contracts starts from a pre-trained [encoder like BERT](https://howaiworks.ai/glossary/large-language-model) — 110M parameters in its base configuration, 340M in its large one, per the original paper — and fine-tunes it on their in-house labels. Starting from those weights routinely reaches usable accuracy on a dataset a thousand times smaller than pre-training used.

Parameter-efficient adaptation has made this even more granular. Because a LoRA adapter is tiny, practitioners keep one frozen base model in memory and swap small task-specific adapters in and out — the community around image models like Stable Diffusion, for instance, shares thousands of LoRA adapters that each teach a shared base model a particular style or subject without anyone re-training the base. One expensive pre-trained artifact, many cheap specializations, is the whole economy of the ecosystem.

## Challenges

The failure mode that catches beginners is treating fine-tuning as free and safe. It is neither, and the two things that break both come from adapting on too little data.

**Catastrophic forgetting** is the first. If you fine-tune every parameter of a pre-trained model hard on a narrow dataset, gradient updates can overwrite the very general knowledge that made the model worth starting from — the network gets better at your 500 examples and measurably worse at everything else. This is exactly why [PEFT](https://howaiworks.ai/glossary/fine-tuning) methods and layer freezing exist: keeping most weights fixed protects the pre-trained knowledge from being trampled by a small task signal. The related term [catastrophic forgetting](https://howaiworks.ai/glossary/catastrophic-forgetting) covers the mechanism in depth.

**[Overfitting](https://howaiworks.ai/glossary/overfitting)** is the second and closely linked. A model with hundreds of millions of parameters can simply memorize a few thousand fine-tuning examples, scoring near-perfect on your training set and failing on anything new. A network trained from scratch on so little data would obviously overfit; the danger with pre-trained models is that strong starting weights *mask* the problem — training loss looks healthy while the model quietly loses generality.

Beyond training dynamics, pre-trained models carry their origins with them. Whatever bias sat in the pre-training corpus ships inside the weights and can survive fine-tuning, so a downstream user inherits problems they never introduced. Licensing is a practical trap of the same kind — "open-weight" does not always mean free for commercial use, and the terms attach to the checkpoint you downloaded, not to your fine-tuned copy. And a pre-trained model is a snapshot of the world at training time: it knows nothing that happened after its data cutoff unless you add retrieval or continue training.

## Future Trends

The clearest trend is that **parameter-efficient adaptation is becoming the default, not the exception.** As base models grow, full fine-tuning of every weight becomes impractical for most teams on cost and memory grounds, while methods like LoRA and its quantized variant QLoRA push the trainable fraction well under 1% and let a single consumer GPU adapt a model that would take a cluster to train. The economic gap this page describes — expensive pre-training, cheap adaptation — is widening, which pushes ever more of the ecosystem toward reusing a small number of heavily pre-trained bases rather than training new ones.

## Frequently Asked Questions

### What is the difference between a pre-trained model and a foundation model?

A foundation model is a large pre-trained model that is general enough to serve many different tasks, so every foundation model is pre-trained but not every pre-trained model is a foundation model. A ResNet trained only to classify ImageNet photos is pre-trained; it is not a foundation model because it does one narrow thing.

### Why not just train a model from scratch?

Pre-training a large model is a one-time megaproject: cost scales with parameters multiplied by training tokens, so a modern model costs enormous compute to train once. Adapting an already-trained model to your task reuses all of that work, which is why almost no one trains from scratch when a suitable pre-trained model exists.

### How much of a pre-trained model does fine-tuning actually change?

It depends on the method. Full fine-tuning updates every parameter, but parameter-efficient methods like LoRA train a tiny fraction. On GPT-3 175B, the LoRA paper reports training as few as 4.7M parameters instead of all 175 billion, roughly 0.003%, while matching full fine-tuning quality.

### Can fine-tuning damage a pre-trained model?

Yes. Fine-tuning on a small dataset can cause catastrophic forgetting, where the model overwrites the general knowledge it learned in pre-training, and overfitting, where it memorizes your few examples instead of learning the task. Freezing most of the network or using parameter-efficient methods reduces both risks.

### Where do pre-trained models come from?

You download them. Public hubs like Hugging Face host millions of pre-trained checkpoints — more than two million as of 2026 — and labs release open-weight models you can fetch and adapt. You start from those weights rather than from random initialization.

## Related

### Related terms

- [Transfer Learning (TL)](https://howaiworks.ai/glossary/transfer-learning)
- [Fine-tuning (FT)](https://howaiworks.ai/glossary/fine-tuning)
- [Foundation Models](https://howaiworks.ai/glossary/foundation-models)
- [Catastrophic Forgetting](https://howaiworks.ai/glossary/catastrophic-forgetting)
- [Self-supervised Learning (SSL)](https://howaiworks.ai/glossary/self-supervised-learning)
- [Overfitting](https://howaiworks.ai/glossary/overfitting)

---

Source: https://howaiworks.ai/glossary/pre-trained-models — HowAIWorks.ai