Meta-Learning

Definition

Meta-learning, or "learning to learn", trains a model on thousands of small learning problems instead of one big one, so that the capability it ends up with is adapting, not any particular task. A meta-learned image classifier is never trained to recognise dogs; it is trained across a stream of tiny five-class problems until five labelled photos of five classes it has never seen are enough to make it work.

The distinguishing mechanism is two nested loops, and one detail inside them decides whether a method is meta-learning at all. The inner loop is ordinary learning on one small task: take its handful of labelled examples (the support set), run one or a few gradient descent steps. The outer loop then grades the adapted model on held-out examples of the same task (the query set) and updates the starting parameters so the next inner loop goes better. So the training loss is measured after adaptation, on data the adaptation never saw. Adaptability is the objective, not a side effect.

That is the whole trick, and it is expensive. In the original Model-Agnostic Meta-Learning (MAML) experiments — Finn, Abbeel and Levine, ICML 2017 — meta-training on the mini-ImageNet benchmark ran 60,000 outer iterations at 4 tasks each, which works out to roughly 19 million image presentations, to produce a model whose selling point is learning a new five-class problem from 5 photos. Meta-learning does not make learning cheap. It moves the cost from adaptation time to training time, and pays it once.

Four things get conflated here constantly, and the difference matters when you are choosing a method:

• Few-shot learning is a problem setting — you get K examples per class and have to work. Meta-learning is one family of answers to it, and not the only one: a plainly fine-tuned backbone is a competitive answer that involves no meta-training at all.
• Transfer learning reuses representations built for a different objective. Nothing during that original training was optimised for how easily the weights would move later; you find out afterwards.
• Fine-tuning is the adaptation step itself. Meta-learning uses it as a subroutine — MAML's inner loop is fine-tuning, differentiated through, with the resulting error backpropagated into the initialisation.
• In-context learning adapts inside a single forward pass with no weight update. GPT-3's authors framed their own system as meta-learning and called in-context learning "the inner loop of this process, which occurs within the forward-pass upon each sequence" — but the outer objective there is next-token prediction, not post-adaptation error, so the adaptability is emergent rather than targeted.

The one-line test that separates them: where is the loss measured? If your training objective is evaluated on a model that has already adapted to a task it will be graded on, you are meta-learning. If it is evaluated on the model as trained, you are not.

How It Works

An episode, counted out

Meta-learning replaces the training batch with the episode, and the vocabulary is N-way K-shot: sample N classes the model has never seen, give it K labelled examples of each, then score it on fresh examples of those same N classes. MAML follows the protocol of Vinyals et al. and, following Ravi and Larochelle, grades each task on 15 query examples per class; prototypical networks use 15 per class too.

So one 5-way 1-shot episode on mini-ImageNet is 5 support images and 5 × 15 = 75 query images — 80 images total, of which the model is allowed to learn from 5 and is graded on 75. A 5-way 5-shot episode is 25 support and 75 query, 100 images. mini-ImageNet itself is 60,000 images at 84×84 across 100 classes, 600 per class, split so that 64 classes are available for meta-training and the rest are held out. Omniglot, the other standard benchmark, is 1,623 handwritten characters from 50 alphabets with 20 instances of each — a dataset built to be wide and shallow rather than deep, because the point is the number of tasks, not the number of examples per task.

The class split is what makes the benchmark honest. The meta-test classes are not merely unseen images, they are unseen categories, so nothing about the specific answer can have been memorised. From 64 meta-training classes there are 7,624,512 distinct ways to pick 5, which is why the task distribution never runs out even though the image pool is small.

The inner loop

For one task, MAML starts from the shared initialisation θ and takes gradient steps on the support set alone. On mini-ImageNet the paper used 5 inner steps with a fixed step size of 0.01 during meta-training, and 10 steps at meta-test time; on 5-way Omniglot, a single inner step with a step size of 0.4 was enough, with 3 steps used at evaluation. Those adapted parameters — often written θ' — are what gets graded.

Nothing in the inner loop is unusual. It is gradient descent, on five images, for five steps. The unusual part is what happens to the error afterwards.

The outer loop, and why it needs second derivatives

The outer loop asks: how should θ change so that the θ' produced from it does better? Because θ' was itself produced by a gradient step on θ, differentiating the query loss with respect to θ means differentiating through the inner update — which contains a gradient. A derivative of a gradient is a second derivative, so the exact meta-gradient requires Hessian-vector products and an extra backward pass, and the memory to hold the unrolled inner trajectory grows with the number of inner steps.

This is the single biggest engineering cost in gradient-based meta-learning, and it is measurable. MAML's authors tested a first-order approximation (now usually called FOMAML) that simply drops the second-derivative term, and reported that removing the Hessian-vector products "led to roughly 33% speed-up in network computation". The accuracy loss was essentially nil: on mini-ImageNet 5-way, full MAML scored 48.70% ± 1.84 (1-shot) and 63.11% ± 0.92 (5-shot), while the first-order version scored 48.07% ± 1.75 and 63.15% ± 0.91 — a difference inside the confidence intervals, and in the 5-shot case pointing the wrong way. The paper's explanation is that ReLU networks are locally almost linear, so the second derivatives are close to zero for much of training.

Reptile (Nichol et al., 2018) pushes the same idea further and needs no query batch to form its meta-gradient at all: run k steps of ordinary SGD on a task to reach weights φ', then move the initialisation a fraction of the way toward them, φ ← φ + ε(φ' − φ). It never computes a second derivative or stores an unrolled graph, and the only extra hyperparameter is the outer step size ε.

What meta-training actually costs

The arithmetic is worth doing once, because it is the honest answer to "why isn't everyone doing this?". For MAML's 1-shot mini-ImageNet result: a meta batch of 4 tasks, 60,000 meta-iterations, 80 images per episode.

• Tasks seen: 4 × 60,000 = 240,000 episodes.
• Images presented: 240,000 × 80 ≈ 19.2 million, against a training pool of 64 × 600 = 38,400 distinct images — roughly 500 exposures per image.
• Inner gradient steps: 5 per task, so 240,000 × 5 = 1.2 million inner updates, on top of 60,000 outer updates, each of which backpropagates through five of those.
• What the finished model then needs to learn a new 5-class problem: 5 images and 10 gradient steps.

That ratio — about 19 million images in, 5 images out — is the trade meta-learning makes. The whole run fitted on a single NVIDIA Pascal Titan X, so the absolute cost was modest; the point is the shape, not the price. If you only ever face one task, this is a spectacularly bad way to spend compute. It pays off when new tasks keep arriving and each must be served from almost no data.

Types

The literature splits cleanly into three families, distinguished by what the outer loop actually learns.

Optimization-based: learn the starting point. MAML learns an initialisation θ whose defining property is that a few gradient steps from it land somewhere good, for any task in the distribution. Meta-SGD extends this by meta-learning a per-parameter learning rate as well; Ravi and Larochelle's meta-learner LSTM learns the update rule itself, with an LSTM playing the part of the optimiser. FOMAML and Reptile are the cheap first-order members of the family. What is learned is a position in weight space, and adaptation is real gradient descent.

Metric-based: learn the embedding, then compare. Prototypical networks (Snell et al., 2017) embed every support example, average the embeddings of each class into a prototype, and classify a query by Euclidean distance to the nearest prototype. There is no inner gradient loop at all — adaptation is one averaging operation, which is why these methods are the cheapest to deploy. On mini-ImageNet the original paper reported 49.42% ± 0.78 (1-shot) and 68.20% ± 0.66 (5-shot), beating MAML's 5-shot number while doing far less work at test time. Matching networks (Vinyals et al., 2016), which classify by attention-weighted comparison to every support example rather than to a prototype, reported 46.6% and 60.0% on the same task in the authors' own experiments. Relation networks replace the fixed distance with a learned comparison module. What is learned here is an embedding function; the "learning" at test time is arithmetic on vectors.

Model-based: learn an update rule that lives in the forward pass. Memory-augmented neural networks (Santoro et al., 2016) attach a Neural Turing Machine-style external memory to a controller and let it learn a read/write policy that binds labels to representations within a single sequence — 82.8% on 5-way 1-shot Omniglot in the results reproduced by MAML's authors. SNAIL combines temporal convolutions with attention to the same end. This family matters far more than its benchmark numbers suggest, because it is the one that won: a transformer doing in-context learning over a prompt of examples is a model-based meta-learner whose inner loop is a forward pass.

Real-World Applications

Meta-learning's industrial footprint is much smaller than its publication count, and the honest version of this section is short. Three uses have genuinely left the lab.

Warm-starting AutoML. auto-sklearn (Feurer et al., NeurIPS 2015) meta-learns across datasets rather than across image classes: offline, it evaluated configurations on 140 datasets from the OpenML repository and characterised each with 38 meta-features; at run time it finds the k = 25 nearest datasets by meta-feature distance and seeds Bayesian optimisation with the configurations that worked there. This is meta-learning in its oldest sense — learning which learner to use — and it is shipped, open-source software that won the ChaLearn AutoML challenges in 2015-2016 and 2017-2018.

Few-shot classification wherever labels are scarce and the domain is fixed. This is where the metric-based family earns its keep, because a prototype-and-distance classifier adds a nearest-neighbour lookup to an encoder you already have and nothing else. Low-data drug discovery is a documented case with a documented limit: Altae-Tran et al. (2017) meta-trained on nine Tox21 assays and tested on three held-out ones, reaching a median score of 0.840 with 10 positive and 10 negative support examples where a 100-tree random forest managed 0.563. Then they trained on Tox21 and evaluated on SIDER — a genuinely different biological question — and the same model returned 0.509, which is chance.

In-context learning in large language models. If you count it, and GPT-3's own authors did, this is by far the largest deployment of the idea: the paper uses "meta-learning" for the inner-outer structure and "in-context learning" for the inner loop. Every few-shot prompt served by a large language model today is an inner loop running in a forward pass, with the context window playing the role of the support set.

What has not happened is a wave of MAML-style two-loop systems in production. The reasons are below, and they are worth reading before you build one.

Key Concepts

• The two halves of an episode: one is what the model may learn from, the other is what it is scored on. Keeping them disjoint is what forces the outer loop to optimise for generalisation instead of memorisation.
• Meta-train / meta-test split at the class level: the held-out data is a set of unseen categories, not unseen examples of familiar categories. Splitting by example instead — an easy mistake in a custom benchmark — inflates every number you will report.
• Where the guarantees come from: meta-learning assumes meta-training and meta-test tasks are drawn from one distribution. Almost everything a benchmark measures depends on that assumption, which is why the cross-domain results below are so much worse than the headline ones.
• Meta-overfitting: overfitting at the level of tasks rather than examples. The model can memorise the shape of the training tasks — the number of classes, their granularity, the imaging conditions — and fail on a task of the same size drawn from anywhere else.

Challenges

The distribution assumption is where it breaks, and the effect is large. Chen et al. (ICLR 2019) built the obvious test the benchmarks had skipped: meta-train on mini-ImageNet, meta-test on CUB bird species, with a ResNet-18 backbone, 5-shot. A plainly fine-tuned baseline scored 65.57% ± 0.70. MAML scored 51.34% ± 0.72 — over fourteen points worse — with matching networks at 53.07% and prototypical networks at 62.02%. The paper's conclusion is blunt: the baseline outperforms every meta-learning method under domain shift, because meta-learners have been trained to expect support sets that look like the ones they were meta-trained on. This is the single most useful fact on this page. If your production tasks will not resemble your meta-training tasks, meta-learning is likely to cost you accuracy relative to fine-tuning, on top of the training bill.

Much of the benefit may not be the adaptation at all. Raghu et al. (ICLR 2020) froze the network body during MAML's inner loop and let only the final classification layer adapt. On mini-ImageNet 5-way 1-shot, accuracy went from 46.9% ± 0.2 with full inner-loop adaptation to 46.3% ± 0.4 with all four convolutional layers frozen. Their ANIL variant — "almost no inner loop" — scored 46.7% ± 0.4 (1-shot) and 61.5% ± 0.5 (5-shot) against MAML's 46.9% ± 0.2 and 63.1% ± 0.4, while running 1.7× faster per training iteration and 4.1× faster at inference. The conclusion is that MAML's success is mostly feature reuse: the meta-initialisation already contains good features, and the fast adaptation everyone was admiring is largely re-fitting a linear head. That reframes meta-learning as an expensive way to learn a representation — and invites the comparison with just learning a good representation directly.

Second-order cost scales with inner steps. Exact meta-gradients require holding the unrolled inner trajectory in memory and computing Hessian-vector products through it, so doubling the inner steps roughly doubles both. Since the first-order approximation matched full MAML to within its confidence intervals for a 33% speedup, the exact gradient is often not worth paying for — but you cannot know that for your task distribution without running both.

Benchmarks that flatter the method. Omniglot is close to saturated: MAML reported 95.8% ± 0.3 on 20-way 1-shot back in 2017, leaving little room to distinguish anything. mini-ImageNet is a narrow, single-source distribution of 84×84 crops, which is exactly the regime where the same-distribution assumption is safest and the cross-domain failure above is invisible. A method tuned on those two benchmarks has been selected for the environment they define.

Task design is the real work, and it is unglamorous. Meta-learning needs a distribution of tasks, which most organisations do not have lying around. Manufacturing one — how many classes per episode, how granular, sampled from where — is a modelling decision that determines the result more than the algorithm does. Prototypical networks make the point sharply: training with 30-way episodes gave better 5-way test accuracy than training with 5-way episodes, so even matching the training task to the test task can be the wrong call.

Future Trends

The field's centre of gravity has already moved, and a forecast that ignored this would misread the last several years.

Large pretrained models absorbed the promise. The pitch was rapid adaptation to new tasks from a few examples, and that is now delivered by foundation models through in-context learning — at a scale no episodic meta-learner reached, using an outer objective, next-token prediction, that nobody designed as a meta-objective. GPT-3's authors described their own structure in the same inner-loop/outer-loop terms MAML used, which is the fair reading: the mechanism survived, the explicit two-loop training procedure mostly did not.

Where it is still the right tool. Meta-learning remains the sensible framing when there is a genuine, stable distribution of tasks and adaptation must happen in weights rather than in a prompt: AutoML and algorithm selection, where auto-sklearn's warm-start is a decade old and still works; on-device personalisation, where each user is a task; reinforcement learning across parameterised environments, where there is no prompt to put examples in. Learned optimisers — meta-learning the update rule itself rather than the initialisation — are the most ambitious surviving line, and remain hard to make robust outside the workloads they were meta-trained on, which is the distribution-shift problem again in a different costume.

The open question is whether the explicit outer loop comes back at scale: a training objective that directly rewarded post-adaptation performance across a distribution of tasks would be meta-learning by the definition at the top of this page, run at foundation-model scale. Nobody has shown that beats pretraining on more data. Until someone does, "learning to learn" is best understood as a concept that explains what large models do, rather than a recipe most teams should adopt.

Code Example

The two-loop structure in the form that shows where the cost comes from. Everything hinges on create_graph=True: it keeps the inner update inside the autograd graph so the outer loss can be differentiated through it.

import torch

meta_opt = torch.optim.Adam(model.parameters(), lr=1e-3)   # outer step size (beta)
meta_opt.zero_grad()

for support, query in sample_tasks(n_tasks=4):             # meta batch: 4 episodes
    fast_weights = list(model.parameters())

    for _ in range(5):                                     # inner loop: 5 steps, alpha = 0.01
        loss = cross_entropy(model(support.x, fast_weights), support.y)
        grads = torch.autograd.grad(loss, fast_weights, create_graph=True)
        fast_weights = [w - 0.01 * g for w, g in zip(fast_weights, grads)]

    # Graded AFTER adaptation, on examples the inner loop never saw.
    meta_loss = cross_entropy(model(query.x, fast_weights), query.y)
    meta_loss.backward()          # backpropagates through all five inner updates

meta_opt.step()

Two lines carry the whole argument. create_graph=True is what makes this second-order: it is why memory grows with the number of inner steps, and dropping it gives you FOMAML at roughly a third less compute. And meta_loss is computed from query, never from support — that is the definition of meta-learning, expressed as one variable name. Change that line to use support and you have written ordinary multi-task fine-tuning with extra steps.

For a metric-based method the same file is far shorter: embed the support images, average them per class, and classify queries by distance. There is no inner loop to differentiate through, which is why prototypical networks are the sensible first thing to try on a real few-shot problem.