Definition
A world model is a learned simulator of how some environment behaves: you feed it the current state and an action, and it predicts the next state — typically the next observation, the reward that action earned, and whether the episode has ended. Because those one-step predictions can be chained together, an agent can roll out an entire imagined future inside the model and choose what to do without ever touching the real world.
That last point is the whole reason the idea exists. Trying an action for real costs time, hardware, and sometimes safety — a robot arm that drops the part, a car that takes the wrong line. Trying it inside a world model costs one forward pass of a neural network. The model turns "act, then see what happens" into "predict what would happen," which you can do millions of times, in parallel, at no physical risk. The catch, which the rest of this page keeps returning to, is that a prediction is only as trustworthy as the model, and a chain of predictions decays fast.
A world model is not the same thing as a video generator, even though the newest ones look like one. A video generator predicts pixels — a plausible continuation of what you were watching. A world model predicts the consequence of an action and exposes a state you can plan over. The difference is action-conditioning and a usable state, and it is the difference between a movie and a flight simulator.
How It Works
Almost every world model has the same three jobs, whatever the architecture underneath.
First, perception: raw observations (say, a 64×64 image) are compressed into a compact latent state, because predicting the future pixel-by-pixel is wasteful and predicting it in a small learned code is tractable. Second, dynamics: a transition model takes the current latent state and a proposed action and predicts the next latent state, along with reward and a done flag. Third, optionally, decoding: the latent can be rendered back into an observation so a human — or the agent — can see the imagined frame. Chain the dynamics model against its own outputs and you get a rollout: an imagined trajectory of states stretching some horizon into the future.
There are two things you can then do with that rollout, and they are the two reasons to build a world model at all.
- Plan. Simulate several candidate action sequences forward, score how each turns out, and execute the first action of the best one — then re-plan at the next step. This is model-predictive control, and it needs no separate learned policy; the model is the plan.
- Train an agent in imagination. Instead of learning a policy from expensive real interactions, learn it from cheap imagined ones. This is model-based reinforcement learning: the policy and value function improve on trajectories the world model dreams up, and only occasionally is the model refreshed with new real data.
The lineage is short and worth knowing. The modern framing comes from Ha and Schmidhuber's 2018 paper "World Models," which paired a variational autoencoder for vision, a recurrent mixture-density network for memory, and a deliberately tiny linear controller. Their striking result: they trained the controller entirely inside the model's own hallucinated dream of the VizDoom "Take Cover" game and then transferred it back to the real game, where it scored around 1,100 time steps against a "solved" threshold of 750 — a policy that had never seen a single real frame during its own training. The Dreamer line (culminating in DreamerV3, 2023) generalized this to learning behaviors purely from imagined rollouts in a learned latent space, across many domains with one fixed configuration.
Here is why "imagine it" beats "try it" in raw numbers. Suppose a policy needs about 1,000,000 environment steps to learn. Collected for real at 10 steps per second, that is 1,000,000 / 10 = 100,000 seconds ≈ 28 hours of continuous operation — per training run, with every clumsy or unsafe action happening physically, and you rarely get a policy in one run. Inside a world model, a step is a batched forward pass: run 1,000 imagined rollouts in parallel and you produce those million steps in a few thousand waves, finishing in minutes and breaking nothing. The saving is not marginal; it is the difference between a research programme and a fleet of crashed robots.
Real-World Applications
Robot and agent control. DreamerV3 is the clearest demonstration that learning in imagination works out of the box: applied without task-specific tuning, it was the first algorithm to collect diamonds in Minecraft from scratch, with no human demonstrations and no curriculum — a long, sparse-reward task that had defeated model-free methods — and it holds up across more than 150 tasks in seven domains under a single set of hyperparameters. The same sample efficiency is what makes world models attractive for robotics, where every real trial wears out hardware and takes wall-clock time that imagined trials do not.
Autonomous-driving simulation. Generative world models are becoming the simulator that self-driving systems are validated against. Wayve's GAIA models (GAIA-1 in 2023) generate realistic driving scenarios conditioned on video, text, and action inputs, so an engineer can prompt "car cuts in from the left in fog" and test how a driving policy responds — a scenario that is dangerous and rare to collect on real roads. This is exactly the coming problem flagged under autonomous vehicle safety: when the world model is the test track, its blind spots become the fleet's, because the scenario the model never generates is the one nobody thought to validate.
Interactive generated environments. Genie (2024), an 11-billion-parameter model, generates action-controllable 2D game worlds: give it a single image and a stream of actions and it produces a playable environment frame by frame. Remarkably, it learned to do this unsupervised from tens of thousands of hours of internet gameplay video with no action labels — it inferred a latent action space itself — and that learned action space can then be used to train agents to imitate behaviors from videos they were never trained on. It is the same machinery as ordinary video generation, turned into something you can steer.
Key Concepts
Action-conditioning is the dividing line. The single question that separates a world model from an impressive video model is: can you inject an action and get back the consequence? A video generator answers "what does a plausible next frame look like?"; a world model answers "what happens if I turn left?" Genie's headline contribution was precisely making generated video controllable action-by-action. Without that hook, generated frames are a movie; with it, they are an environment.
Latent space versus pixel space. Rolling the future forward in a compact latent code, as Dreamer does, is far cheaper than rendering every imagined frame in full resolution, and it tends to generalize better because the model is forced to keep only what matters for prediction. Pixel-space rollouts are more interpretable — you can watch the dream — but expensive and prone to drifting into visual nonsense over a long horizon.
Model-based versus model-free RL. A model-free agent learns a policy directly from real reward signals; it is simple and robust but data-hungry. A model-based agent spends effort learning the world first and is rewarded with dramatic sample efficiency — at the cost of a new failure mode: the policy is now only as good as the model it dreamed in. See reinforcement learning for the broader framing.
Challenges
Compounding error over the horizon. This is the fundamental limit, and it is worth doing on numbers. Suppose the one-step predictor is 99% reliable — 99 times in 100, its predicted next state stays within tolerance. Errors do not reset each step: step t+1 is predicted from the already-slightly-wrong predicted state at t, so inaccuracies carry forward. Treating each step as an independent chance to stay on track, the probability an H-step rollout is accurate end-to-end is roughly 0.99 raised to the H:
- 10 steps: 0.99^10 ≈ 0.90
- 50 steps: 0.99^50 ≈ 0.61
- 100 steps: 0.99^100 ≈ 0.37
- 200 steps: 0.99^200 ≈ 0.13
A model that looks excellent by its single-step error — 99% correct — is trustworthy for only about 37% of 100-step rollouts. And 0.99 is generous: at 0.95 per step, 0.95^100 ≈ 0.006, essentially never. Worse, real dynamics compound faster than this independence assumption, because errors feed back — a wrong predicted state leads the agent to a different action, which pushes the next predicted state further off. This is why practical systems keep planning horizons short and periodically re-ground imagined rollouts in real observations rather than dreaming indefinitely.
Hallucinated dynamics. A world model can invent physics that does not hold, and an agent trained inside it will happily exploit the fiction. Ha and Schmidhuber saw this directly: their controller found ways to game imperfections in the dreamed Doom — where, for instance, monsters' fireballs would sometimes flicker out of existence — learning a policy that thrived in the dream and would have failed in reality. Their fix was to raise the model's sampling temperature so the dream became more uncertain and harder to exploit, forcing a policy robust enough to survive the real thing. The lesson generalizes: a world model that is too easy to fool teaches an agent the wrong lessons, confidently.
Distribution blind spots. A model only generates what its training distribution taught it to generate. Rare, safety-critical events — the pedestrian in an unusual pose, the debris in the lane — are exactly the ones under-represented in the data and therefore under-generated by the model. When the world model is used as a validation environment, that gap is silent and dangerous, because a scenario the model cannot produce is a scenario the tested policy is never asked about.
Future Trends
The clear direction is toward large, general foundation world models — the same scaling story that produced foundation models in language, now applied to learned environments. Rather than a leaderboard of specific systems (which turns over quickly), think of it as a category: interactive, promptable simulators trained on vast video corpora, of which Genie and the GAIA driving models are early public examples. Two currents are converging on it — the video-generation community arriving at controllability, and the reinforcement-learning community arriving at scale — and where they meet is a model you can both watch and act inside.
The stakes rise as these models move from research demos to the thing safety-critical systems are validated against. If the future of testing an autonomous vehicle, a warehouse robot, or an embodied agent is largely inside a world model, then the accuracy, coverage, and honesty of that model become a safety property in their own right — which makes the compounding-error and blind-spot problems above not academic caveats but the central engineering challenge of the next few years.
Note: this term was last reviewed in July 2026. The specific systems and figures cited — Ha and Schmidhuber's 2018 results, DreamerV3 (2023), Genie (2024, 11B parameters), and Wayve's GAIA (2023) — are stated as of that review and are the parts of this page most likely to age; the compounding-error arithmetic is not.