Definition
Simple Continual Pretraining (SCP) is a recipe for converting a pretrained autoregressive (AR) language model into a diffusion language model: you replace the model's causal attention mask with a bidirectional one at the very start, then keep pretraining it with the masked-diffusion objective. The point is that you do not train the new model from scratch — you inherit all the world knowledge the AR model already absorbed from trillions of tokens, and spend a comparatively small budget teaching it to generate in parallel instead of left-to-right.
The method was named and described in Radical Numerics' October 2025 report on RND1, where a single SCP run of 500B tokens turned the autoregressive Qwen3-30B-A3B into a 30B-parameter diffusion model that led open diffusion language models on standard benchmarks. "Simple" is the operative word: the recipe is one stage with no gradual mask schedule, which is what makes it easy to reproduce and to scale.
How It Works
To see why the conversion is even possible, start with what the two model families share. An autoregressive model predicts the next token given everything to its left; a diffusion language model predicts many masked tokens at once, in flexible order, from the surrounding context. Predicting a masked token from both sides is a generalization of predicting the next token from the left, so an AR model is already most of the way to being a diffusion model — it just needs to learn to read context in both directions. SCP exploits exactly that overlap.
The single-stage recipe
The AR checkpoint uses a causal mask, a triangular matrix that forbids each position from attending to tokens that come after it. SCP replaces that mask with a bidirectional mask at initialization, so every position can now attend to the whole sequence. Training then continues with the masked-diffusion loss — a fraction of tokens are hidden and the model is scored on reconstructing them — under a learning-rate warmup for stability. There is no separate "AR phase", no annealing, and no architecture surgery; the mask change and the new objective are switched on together at token zero of the continued run.
That simplicity is the design claim. The RND1 report contrasts SCP with two heavier alternatives it benchmarked against. Training from scratch (random initialization) throws away the AR head start and, in their 4B-scale probe, kept the highest loss throughout. Grafting first trains a causal diffusion model, then transplants bidirectional attention operators and continues — a multi-step procedure. At 20B tokens of probing on Qwen3-4B, SCP roughly matched grafting and both beat from-scratch training, so the extra machinery bought little. These are alternative conversion strategies rather than flavors of SCP, which is why SCP itself has no sub-types.
Keeping the old knowledge with grouped learning rates
The hard part is not adding bidirectional attention — it is not destroying the factual knowledge the AR model already holds while you do it. Update every weight aggressively and the model drifts: in RND1's ablation, a single uniform learning rate made GSM8K accuracy fall monotonically as conversion went on, a textbook case of catastrophic forgetting.
SCP's fix is to set separate peak learning rates by parameter group. In the recipe RND1 adopted, attention parameters get a peak rate of 3×10⁻⁴ so they can learn the new bidirectional patterns, while everything else — the feed-forward and Mixture-of-Experts weights, embeddings, normalization, and routers — gets a near-zero 1×10⁻⁸, with weight decay of 0.1 across the model. Because the bulk of a Transformer's learned facts live in its feed-forward weights, freezing those in place while retraining attention lets the model gain bidirectionality with almost no measured knowledge loss. An intermediate setting (attention at 1×10⁻⁴, the rest at 1×10⁻⁶) forgot less than a uniform rate but learned slowly, which is the trade-off you are tuning: the closer the non-attention rate is to zero, the more knowledge you keep and the less the model adapts.
Diffusion wants bigger batches
One more recipe detail matters at scale. In an AR model every token in a sequence contributes to the loss; in a masked-diffusion model only the masked positions do, so each batch delivers a weaker learning signal. That changes the batch-size arithmetic. RND1 estimated the critical batch size — the point past which adding more data-parallel batch stops helping — by branching one run into copies at 1M, 2M, 4M, and 8M tokens per step. Loss kept dropping all the way to 8M, meaning the critical batch size for the diffusion run lay beyond 8M tokens: diffusion conversion tolerates, and benefits from, larger batches than the AR heuristics would suggest. The small-scale probes used a 2M-token global batch; the larger conversion runs used a 33.5M-token batch.
Real-World Applications
The concrete, checkable use of SCP is RND1-Base, the diffusion language model Radical Numerics released with the recipe in October 2025. It was produced by running SCP on Qwen3-30B-A3B for 500B tokens, yielding a 30B-parameter Mixture-of-Experts diffusion model with about 3B parameters active per token. The report presents it as the first open effort to push diffusion language models past 8B parameters, and releases the weights, inference code, and recipe.
Against the prior open diffusion baselines it beat both on every benchmark reported. On general reasoning it scored 69.6% on MMLU and 67.5% on BBH; on math, 80.0% on GSM8K; on coding, 65.4% on MBPP — ahead of Dream-7B (57.9% BBH, 77.2% GSM8K) and LLaDA-8B (47.4% BBH, 70.9% GSM8K) on those same rows. The honest ceiling is also visible in the same table: RND1's autoregressive parent, Qwen3-30B-A3B, still scored higher (79.5% MMLU, 85.2% GSM8K), so the conversion narrows but does not close the gap to the AR model it came from.
Why go to the trouble at all? A diffusion language model generates tokens in parallel and can revise them across denoising steps rather than committing to a fixed left-to-right order, which opens different inference-time trade-offs than an autoregressive pre-trained model. SCP is the practical on-ramp: it lets a lab reuse a mature AR foundation model and its training infrastructure instead of standing up a diffusion pretraining pipeline from nothing.
Challenges
The conversion cost is real, not free. "Continual" pretraining still meant 500B tokens on a 64-GPU cluster for RND1 — orders of magnitude more compute than a typical fine-tuning run, even though it is far cheaper than pretraining a diffusion model from scratch. SCP moves the cost, it does not remove it.
The retention knob is a balance, and mis-setting it fails in either direction. Push the non-attention learning rate too high and knowledge erodes; push it to near-zero and adaptation slows, so the model needs more tokens to converge. There is no free setting that both preserves everything and learns quickly, and the report's four-way ablation exists precisely because the right point is not obvious in advance.
A single large run leaves little room to tune. The RND1 report notes the 30B conversion was executed once. Most of the recipe's hyperparameters — the exact learning-rate ratio, the batch size, the token budget — were fixed from smaller 4B-scale probes on Qwen3-4B and then trusted at 30B, because a second full run is expensive. Whether those choices transfer cleanly to much larger or differently-shaped models is not yet established.
You still inherit diffusion's evaluation headaches. Confirming that a converted model has genuinely become bidirectional, and comparing it fairly against both its AR parent and other diffusion models, is harder than reading a single perplexity number — the generation process, masking ratio, and decoding schedule all affect the scores.
Code Example
The two mechanical changes SCP makes are the mask swap and the grouped learning rates. This illustrative PyTorch-style sketch shows both — an attention layer built to accept either mask, and an optimizer that gives attention weights a normal rate while nearly freezing everything else:
import torch
def attention(q, k, v, bidirectional):
scores = q @ k.transpose(-2, -1) / q.size(-1) ** 0.5
if not bidirectional:
# AR: mask out every position to the right (causal)
seq = scores.size(-1)
causal = torch.triu(torch.ones(seq, seq), diagonal=1).bool()
scores = scores.masked_fill(causal, float("-inf"))
# SCP flips this flag to True at token zero of the converted run
return torch.softmax(scores, dim=-1) @ v
def scp_optimizer(model, attn_lr=3e-4, other_lr=1e-8, wd=0.1):
attn, other = [], []
for name, p in model.named_parameters():
(attn if "attn" in name else other).append(p)
# attention adapts; feed-forward / MoE / embeddings stay near-frozen
return torch.optim.AdamW(
[{"params": attn, "lr": attn_lr},
{"params": other, "lr": other_lr}],
weight_decay=wd,
)
The training loop is otherwise ordinary continued pretraining, except the loss is computed only over masked positions rather than over every next token — which is the reason, as above, that diffusion conversion prefers a larger batch than the AR recipe would use.
Related Concepts
SCP sits next to a few ideas worth separating from it. It resembles fine-tuning in that both continue training from an existing checkpoint, but fine-tuning keeps the architecture and objective and adapts to a task on a small dataset, whereas SCP changes the attention pattern and the objective and needs a large pretraining-scale budget. It shares the anti-forgetting goal of continual learning, but continual learning is about absorbing a stream of new data over time, while SCP is a one-off change of what the model is — from an autoregressive predictor to a diffusion one.
Learn more about related concepts: Diffusion Language Models, Pre-trained Models, Fine-tuning, Foundation Models, and Catastrophic Forgetting.