Definition
Cloud computing is renting computers by the hour instead of owning them — but the three properties that made that promise work for web applications do not hold for AI accelerators. Classic cloud sold CPU that was fungible (one second is like any other), elastic (summon a thousand, release them a minute later) and on-demand (capacity is simply there). A block of a thousand GPUs that must sit physically adjacent on one fast interconnect for six uninterrupted weeks is none of those things. It is scarce, it is committed months in advance, and it is billed as a standing reservation rather than a purchase.
That distinction is what breaks budgets. A team that plans an AI system with classic-cloud instincts assumes capacity will be available when the project is ready and that it pays only for what it uses. Both assumptions are false for accelerators, and the second one is false by a factor of two or three.
How It Works
The unit of rental became the reservation
Elasticity in classic cloud was never magic; it was statistical multiplexing. A provider can safely sell more capacity than it owns because tens of thousands of small tenants peak at different moments, and the aggregate is smooth. A distributed training job destroys that arithmetic. It does not have peaks to smooth — it needs every accelerator simultaneously, for the whole run, and if one node dies the other 16,383 sit idle waiting for a restart. With nothing to multiplex, the provider stops selling you an illusion of infinite machines and starts selling you the machines: one-year and multi-year commitments, capacity blocks reserved for named dates, and increasingly contracts written against a future data center build rather than a pool that already exists.
Adjacency: a GPU is no longer a GPU
Inside one H100 server, NVLink 4 gives each GPU about 900 GB/s to its neighbours. Between servers, a typical 400 Gb/s cluster link is about 50 GB/s per GPU — roughly 18x less. Since every training step exchanges gradients across the whole job, where the accelerators physically sit changes throughput more than which accelerator you chose. "Any twelve idle GPUs in the region" is not a substitute for twelve in one pod, which is why accelerator capacity is sold as located blocks and not as interchangeable instances.
Utilisation is the entire economics
A reservation bills continuously. One thousand accelerators held for a year is
1,000 x 8,760 = 8.76 million accelerator-hours whether or not anything runs on them; at 40% busy,
5.3 million of those hours are paid idle. Effective cost per unit of useful work is the quoted
rate divided by utilisation — 1/0.4 = 2.5x at 40%, 1/0.8 = 1.25x at 80%. Nobody negotiates a
2x discount. Everybody can lose 2x to a badly scheduled queue.
Even a well-run cluster does not reach 1.0. Meta reported 38–43% model FLOPs utilisation in BF16
while pre-training Llama 3, alongside better than 90% effective training time — two different
utilisation numbers, both below one, multiplying together. Put that into the standard 6ND training
estimate (FLOPs ≈ 6 x parameters x tokens). A 70B model on 15 trillion tokens is
6 x 7e10 x 1.5e13 = 6.3e24 FLOPs. At an H100's roughly 1e15 dense BF16 FLOP/s that is 1.75
million GPU-hours if the silicon never stalls; at 40% MFU you rent about 4.4 million, which
spread across 8,192 GPUs is roughly 534 hours — 22 days of wall clock you are paying for from
the moment the reservation starts.
Types
The classic three service models still describe the menu, with a fourth that AI added:
- IaaS — you rent the machine and its network placement, and you own the drivers, the scheduler and every idle hour.
- PaaS — managed training and serving platforms (Vertex AI, SageMaker, Azure AI Foundry) that hide the cluster but not the reservation.
- SaaS — finished applications where the model is an implementation detail.
- Model-as-a-service — a hosted API billed per token rather than per hour. This is the genuinely new one: it moves utilisation risk to the provider, who multiplexes your bursty traffic against everyone else's and can therefore restore the elasticity that raw accelerators lost.
Real-World Applications
Checkpointing cadence is a cloud decision, not a training detail. In a 54-day pre-training
snapshot on 16,384 H100s, Meta logged 466 job interruptions, 419 of them unexpected — about one
every three hours. That is not unreliable hardware: 16,384 x 54 / 419 works out to roughly 2,100
GPU-days, or 5.8 GPU-years between failures for any individual chip. The cluster fails
constantly because it is large, and doubling it halves the time between interruptions. Any rented
cluster past a few thousand accelerators must therefore be architected around restart, not around
uptime.
Where the data already is decides where the compute goes. A 15-trillion-token corpus is roughly
60 TB of text at about 4 bytes per token; sustaining 10 Gb/s, copying it between providers takes
60e12 x 8 / 1e10 ≈ 48,000 seconds — over 13 hours of pure transfer before egress billing.
Google dropped exit fees in January 2024 and AWS followed that March, and the EU Data Act bans
profit-making switching charges from 12 January 2027 — but those cover a full exit, not the routine
cross-provider traffic a multi-cloud pipeline generates every day.
Training and inference pull deployment in opposite directions. Training is a synchronous burst that cares about interconnect and cheap power and can run wherever both exist. Serving is a steady, latency-sensitive load that must sit near users: light in fibre covers about 200,000 km/s, so a Frankfurt–Virginia round trip has a 66 ms floor before any routing overhead — comparable to two or three tokens of generation time, added to every request, fixable only by moving the deployment and not by buying faster chips.
Challenges
Scale-to-zero barely works. A 70B model in BF16 is 140 GB of weights. Loading that into accelerator memory takes about 14 seconds at 10 GB/s from local NVMe and over a minute from object storage — against a web container of perhaps 100 MB that cold-starts in well under a second. So "serverless" AI endpoints mostly keep weights resident, and you pay for idle accelerators under a name that suggests you are not.
Reservation terms outlive hardware relevance. Accelerator generations arrive roughly every two years while commitments run one to three, so a contract signed today will spend part of its term on silicon that is a generation behind — and the energy per useful token on that older hardware is worse too.
Quota, not the API, is the real ceiling. The endpoint that creates an eight-GPU instance is identical to the one that creates a small VM; what differs is whether the request is granted. Teams plan scale against documented limits and discover the binding one is an account quota negotiated with a salesperson.
Future Trends
Capacity contracts are moving upstream toward power: the scarce input is megawatts at a substation, so buyers increasingly sign against a data center's grid connection years ahead. Serving is splitting in the opposite direction — prefill and decode disaggregated onto different hardware pools because they stress compute and memory differently (see inference optimization), with the latency-critical tail pushed toward edge AI. And per-token pricing keeps abstracting the accelerator away entirely, which is convenient right up to the moment a data-residency rule dictates which country the weights may sit in, and hardware becomes visible again.