Definition
Self-supervised learning (SSL) trains a model on unlabeled data by having the data supervise itself: you hide part of each input and make the model predict the hidden part, so the "label" is generated for free from the data's own structure. Because no human has to annotate anything, you can train on billions of examples nobody labeled — and that is exactly the trick that made modern large language models possible, since predicting the next word in raw internet text is self-supervised.
The point is to escape a bottleneck. Supervised learning is powerful but needs a human to attach the correct answer to every training example, and hand-labeling is slow, expensive, and often the hard limit on how much data you can use. SSL sidesteps this entirely by inventing the label from the input. Take a sentence, delete a word, and you have a labeled example: the input is the sentence-with-a-hole, the answer is the word you deleted — and you never paid anyone to write it down. Do that to every sentence on the internet and you have a training set larger than any labeling budget could ever buy.
It helps to place SSL against its neighbors. Supervised learning has human labels; classic unsupervised learning has no prediction target at all — methods like clustering just look for structure. SSL sits in between: no human labels, but a concrete prediction target it built itself. For that reason it is usually described as a subset of unsupervised learning — it uses unlabeled data — that behaves like supervised learning, because internally it is still predicting an answer and getting scored against it.
How It Works
The artificial prediction problem you invent is called a pretext task. You do not actually care about the answer to the pretext task — nobody needs a model whose job is to fill in deleted words. You care about the representations the model is forced to build in order to solve it. To predict a missing word well, a network has to encode grammar, meaning, and world knowledge; those internal features are the real product, and you keep them and throw the pretext task away. This is why SSL is the standard first stage of building pre-trained models: pretrain on a pretext task over huge unlabeled data, then reuse the learned features for whatever you actually want.
The three pretext tasks that matter most in practice are worth seeing concretely.
Masked language modeling (MLM) is how BERT is trained. In the original paper (Devlin et al., 2018), the recipe is precise: "we mask 15% of all WordPiece tokens in each sequence at random," and the model must reconstruct them from the words on both sides. But you cannot simply replace every chosen token with a [MASK] symbol, because [MASK] never appears when you later use the model on real text — the model would learn a habit it can never use. So the 15% is split: "the [MASK] token 80% of the time," "a random token 10% of the time," and "the unchanged i-th token 10% of the time." Predicting a token that was left unchanged forces the model to build a real representation of every position rather than only reacting to the mask symbol. Learning this way, BERT reaches 110M parameters in its base configuration and 340M in its large one — trained on plain unlabeled text, no annotation anywhere.
Next-token prediction is the GPT-family pretext task and the one that scaled furthest. The model reads a stretch of text and predicts the very next token; the label is simply the token that actually came next, already sitting in the corpus. A document of 1,000 tokens is therefore roughly 1,000 free training examples — every position is a prediction with a built-in answer. Because the supervision is that cheap and that abundant, next-token prediction is what let pretraining run over trillions of tokens of web text, and it is the literal training objective underneath today's large language models.
Contrastive learning is the dominant SSL approach in vision, where "predict the missing pixels" is a blunt target. Instead of predicting content, you learn by comparison. SimCLR (Chen et al., 2020) takes one image, applies two different random augmentations — a crop, a color shift — to produce two views, and trains the network to pull those two views' embeddings together while pushing apart the embeddings of every other image in the batch. The "label" is again free and structural: two views of the same photo should agree, two views of different photos should not. The negatives come from the batch itself, and the arithmetic is simple — a batch of N images produces 2N augmented views, so each view is contrasted against the other 2N − 2 as negatives. That is why contrastive methods famously want very large batches: a bigger batch is more negatives to push against, and more negatives sharpen the representation.
Types
Almost every SSL method falls into one of three families, and the split is a real distinction practitioners use, not a made-up taxonomy. They differ in what the model predicts and where — in the input's own space, or in a learned representation space.
- Generative / masked-prediction methods reconstruct a hidden or future part of the input in its own space. Masked language modeling (predict deleted words), next-token prediction (predict the following token), and masked image modeling — reconstruct hidden image patches, the approach the 2021 Masked Autoencoder (MAE) established for vision — all live here. The supervision is a literal piece of the data you withheld.
- Contrastive / joint-embedding methods never reconstruct anything. They learn by comparing examples in embedding space — same-source views pulled together, different-source examples pushed apart. SimCLR and MoCo are the canonical vision examples; non-contrastive variants such as BYOL and DINO reach the same goal without an explicit pool of negatives.
- Joint-embedding predictive methods (JEPA) sit between the two: they predict, but in representation space rather than pixels. I-JEPA (Assran et al., 2023) predicts, from a single context block of an image, the representations of hidden target blocks — no pixel reconstruction and no negatives, which lets the target encoder drop the fine detail a pixel loss would force it to keep.
The line matters because the three families fail and cost differently: generative methods need a way to hide part of the input; contrastive methods need a supply of negative examples (or a trick to avoid them), which drives their appetite for large batches; and joint-embedding predictive methods have to keep their target encoder from collapsing to a constant without either a reconstruction loss or negatives to anchor it.
Real-World Applications
The single biggest application is the pretraining stage of essentially every modern foundation model. GPT-style models are pretrained by next-token prediction over web-scale text; BERT and its descendants by masked language modeling; and that pretrained network is then fine-tuned for translation, classification, search, or chat. When people say a model was "trained on the internet," the mechanism that made unlabeled internet text usable is self-supervised learning.
In computer vision, contrastive and masked-image methods pretrain a backbone on unlabeled images so that downstream tasks — classification, detection, segmentation — can be fine-tuned from strong features with only a small labeled set. This is the same "download, then adapt" economy that pretrained models run on, except the pretraining used no labels at all.
Cross-modal SSL is the third big deployment. CLIP is trained self-supervised on image–caption pairs already present on the web — no human labeled the pairs, the pairing is the supervision — and the result powers text-to-image search and conditions image generators. Speech models such as wav2vec 2.0 do the same for audio, pretraining on unlabeled speech by predicting masked segments of the waveform, then fine-tuning on a fraction of the transcribed data that supervised speech recognition would otherwise require.
Challenges
The failure mode that defines SSL is the shortcut: a pretext task the model can solve without learning anything useful. If your task is "predict the rotation of this image" but many photos share a visible watermark or a consistent sky-up bias, the network can read the shortcut and score well while learning nothing about objects. Designing a pretext task whose only good solution is a good representation is the central craft of the field, and it is genuinely hard — a task that is too easy is trivially gamed, and a task that is too hard produces no useful gradient.
Contrastive methods have a specific version of this called collapse: if nothing pushes different images apart, the network can map every input to the same vector and drive its loss toward the floor while destroying all information. Preventing collapse is exactly what the negatives are for — which is why these methods need many of them, and why the appetite for large batches is a real cost rather than a tuning detail.
Evaluation is indirect, and this trips people up. There is no label to score against, so a low pretext loss does not tell you the representation is good. You find out only by attaching a downstream probe — typically training a simple linear classifier on the frozen features — and measuring that. The signal you actually care about is one step removed from the number the training loop is optimizing.
Finally, the right pretext-task settings do not transfer across modalities. Text needs only a small fraction of tokens hidden to make masked language modeling non-trivial — BERT's 15% — whereas images are so locally redundant that you must hide most of the picture before predicting the rest is hard enough to teach anything. A masking rate copied from one domain to another is a classic way to get a model that trains cleanly and learns little.
Future Trends
Masked-image modeling is no longer the emerging idea — it is the settled one. The 2021 Masked Autoencoder showed that hiding most of an image (roughly 75% of patches) and reconstructing the rest is a strong pretext task that needs no negatives and no giant batches, and vision SSL absorbed it as a default alongside contrastive methods rather than replacing them outright. The live question is no longer "comparison or reconstruction" but what you should predict into.
That is where joint-embedding predictive architectures now draw the most attention. Instead of reconstructing pixels (generative) or matching augmented views (contrastive), I-JEPA (Assran et al., 2023) predicts the representations of hidden regions in a learned space, and the same recipe has since been carried from images to video. The bet is that predicting in representation space forces a model to capture the semantics of a scene while ignoring the pixel-level detail a reconstruction loss wastes capacity on — the direction Yann LeCun and others have argued is the path toward models that learn how the world works, not just how it looks.
The other open front is multimodal self-supervision. CLIP showed that naturally co-occurring pairs — an image and its caption, a video and its audio — are a free supervision signal, because the pairing is already in the data. Extending that across more modalities at once, so one model learns joint representations of text, images, audio, and more with none of it hand-labeled, is now the backbone of how frontier foundation models are built.