Definition
Video generation is the synthesis of moving images from a text prompt, a still image, or another clip. It differs from image generation in one respect that turns out to dominate everything else: the output has to be consistent with itself over time.
The scale of that constraint is worth making concrete. A single 1080p frame is 1920 × 1080 × 3 ≈ 6.2 million numbers. Five seconds at 24 frames per second is 120 frames, so a short clip is roughly 750 million numbers that must agree with one another — the same face on frame 1 and frame 120, the same shirt, a cup that stays on the table and does not drift across it. Generate each frame independently and you get a flicker reel. That is why video models do not generate frames; they generate the whole clip as one object.
No model works on those raw numbers. The video is first compressed into a latent space — downsampled spatially and in time — so that the generative model operates on a representation orders of magnitude smaller, and a decoder expands the result back into pixels at the end. Essentially every text-to-video system in use is built this way, and most of the visible artefacts come from that compression, not from the model's understanding of the prompt.
How It Works
Compression into latents. An autoencoder maps the clip into a compact latent representation, typically downsampling each spatial dimension by a factor of 8 and time by a factor of 4 or more. The generative work happens entirely in that space.
Denoising, jointly across space and time. The model starts from noise in latent space and removes it in steps, conditioned on the prompt. The critical part is that the denoising operates over the whole spatio-temporal volume at once: attention connects a patch in one frame to patches in other frames, which is how the model keeps an object's identity stable rather than re-inventing it every frame. Architectures that treat the video as a sequence of space-time patches, as Sora does, are the current mainstream because they let the same transformer machinery handle both dimensions.
Decoding and upscaling. The finished latent is decoded to pixels, often at a lower resolution than delivered, with a separate upscaling pass afterwards. Splitting it this way is a cost decision: the expensive generative steps run on the small representation.
Conditioning beyond text. Production tools rarely start from text alone. Image-to-video fixes the first frame so the composition is under the user's control; video-to-video restyles existing footage; camera and motion controls constrain the trajectory. The gap between demo prompts and professional use is mostly this — practitioners want to specify, not to describe and hope.
Real-World Applications
Advertising and social content, which is where the money is. Generated video is used at volume for product spots, localised ad variants and social media assets — work with short runtimes, no continuity requirements across shots, and a budget that never justified a shoot. This is the segment where the technology is unambiguously in production rather than in pilot.
Pre-visualisation in film and television. Directors and agencies use generated clips to test a shot, a lighting idea or an edit before committing a crew. The output does not need to be final-quality; it needs to exist tomorrow instead of in three weeks. Runway in particular built its business on professionals using it this way.
Visual effects and clean-up. Inpainting an object out of a moving shot, extending a background plate, or generating an element that would otherwise be a modelling job. This is unglamorous and widely adopted, because it slots into an existing pipeline rather than replacing one.
Education and explanation. Generating a visual for a process — a cell dividing, a mechanism operating — that would otherwise require an animator.
And the misuse case, which is not hypothetical. The same capability produces convincing footage of real people saying and doing things they did not, which is why deepfakes and video generation are the same technology described from two directions.
Leading systems as of this writing include Sora 2 from OpenAI (clips up to 20 seconds, sora-2 at 720p and sora-2-pro at 1080p, with OpenAI having announced that the Sora 2 models and Videos API shut off on 24 September 2026), Kling, Runway and Luma Dream Machine. Specific model capabilities in this field turn over in months; the constraints described above have not.
Challenges
Physics is approximated, not simulated. The model learned what falling water and breaking glass look like, not how they work. Results are convincing for familiar motion and fail on interactions the training data under-represents — a poured liquid whose volume is not conserved, a limb that passes through an object, a gymnast whose limbs reassemble mid-rotation. There is no equation anywhere in the system to violate, which is why these errors are strange rather than merely wrong.
Consistency across shots. Within a clip, temporal attention holds a character together. Across two separately generated clips there is no mechanism doing so at all, and a face, a costume or a room will differ. This — not clip length — is the wall between generated video and narrative filmmaking, and it is why the tooling effort has moved towards reference-conditioning and character locking.
Cost. Generation is compute-heavy in a way image generation is not, and it scales worse than linearly with duration and resolution. Long-form generated video is limited by economics as much as by quality.
Copyright and consent. Training corpora contain copyrighted film and television, and the legal position is unsettled in most jurisdictions. Separately, generating a recognisable person requires their consent under a growing number of laws and under the terms of most commercial providers.
Provenance. Detecting generated video after the fact is a losing race. The industry response is to sign the origin into the file at creation time — see content provenance for the C2PA standard now embedded by the major providers.
Future Trends
Two directions look durable. The first is controllability: camera paths, character references, editable regions and per-shot continuity, all aimed at the professional who needs a specific thing rather than a plausible one. The second is duration bought through architecture rather than brute force — models that generate a long sequence in coherent segments while carrying identity forward, instead of extending the attention window until the cost becomes absurd. Both point at the same destination, which is generated video becoming a component inside an editing workflow rather than a slot machine that returns a finished clip.