--- source: 'https://howaiworks.ai/glossary/cloud-computing' section: glossary title: Cloud Computing description: >- Cloud computing is the on-demand delivery of computing power, database storage, applications, and other IT resources via the internet with pay-as-you-go. tags: - AI Infrastructure - Data Centers - Scaling - Distributed Computing - GPU category: Infrastructure & Hardware datePublished: '2026-01-30' lastUpdated: '2026-07-22' --- # Cloud Computing > Cloud computing is the on-demand delivery of computing power, database storage, applications, and other IT resources via the internet with pay-as-you-go. ## Definition **Cloud computing is renting computers by the hour instead of owning them — but the three properties that made that promise work for web applications do not hold for AI accelerators.** Classic cloud sold CPU that was *fungible* (one second is like any other), *elastic* (summon a thousand, release them a minute later) and *on-demand* (capacity is simply there). A block of a thousand [GPUs](https://howaiworks.ai/glossary/gpu-computing) that must sit physically adjacent on one fast interconnect for six uninterrupted weeks is none of those things. It is scarce, it is committed months in advance, and it is billed as a standing reservation rather than a purchase. That distinction is what breaks budgets. A team that plans an AI system with classic-cloud instincts assumes capacity will be available when the project is ready and that it pays only for what it uses. Both assumptions are false for accelerators, and the second one is false by a factor of two or three. ## How It Works ### The unit of rental became the reservation Elasticity in classic cloud was never magic; it was **statistical multiplexing**. A provider can safely sell more capacity than it owns because tens of thousands of small tenants peak at different moments, and the aggregate is smooth. A [distributed training](https://howaiworks.ai/glossary/distributed-training) job destroys that arithmetic. It does not have peaks to smooth — it needs every accelerator simultaneously, for the whole run, and if one node dies the other 16,383 sit idle waiting for a restart. With nothing to multiplex, the provider stops selling you an illusion of infinite machines and starts selling you the machines: one-year and multi-year commitments, capacity blocks reserved for named dates, and increasingly contracts written against a future [data center](https://howaiworks.ai/glossary/ai-data-center) build rather than a pool that already exists. ### Adjacency: a GPU is no longer a GPU Inside one H100 server, NVLink 4 gives each GPU about **900 GB/s** to its neighbours. Between servers, a typical 400 Gb/s cluster link is about **50 GB/s** per GPU — roughly **18x less**. Since every training step exchanges gradients across the whole job, where the accelerators physically sit changes throughput more than which accelerator you chose. "Any twelve idle GPUs in the region" is not a substitute for twelve in one pod, which is why accelerator capacity is sold as located blocks and not as interchangeable instances. ### Utilisation is the entire economics A reservation bills continuously. One thousand accelerators held for a year is `1,000 x 8,760 = 8.76 million` accelerator-hours whether or not anything runs on them; at 40% busy, **5.3 million of those hours are paid idle**. Effective cost per unit of useful work is the quoted rate divided by utilisation — `1/0.4 = 2.5x` at 40%, `1/0.8 = 1.25x` at 80%. Nobody negotiates a 2x discount. Everybody can lose 2x to a badly scheduled queue. Even a well-run cluster does not reach 1.0. Meta reported **38–43% model FLOPs utilisation** in BF16 while pre-training Llama 3, alongside better than **90% effective training time** — two different utilisation numbers, both below one, multiplying together. Put that into the standard `6ND` training estimate ([FLOPs](https://howaiworks.ai/glossary/flops) ≈ 6 x parameters x tokens). A 70B model on 15 trillion tokens is `6 x 7e10 x 1.5e13 = 6.3e24` FLOPs. At an H100's roughly `1e15` dense BF16 FLOP/s that is 1.75 million GPU-hours *if the silicon never stalls*; at 40% MFU you rent **about 4.4 million**, which spread across 8,192 GPUs is roughly **534 hours — 22 days of wall clock** you are paying for from the moment the reservation starts. ## Types The classic three service models still describe the menu, with a fourth that AI added: - **IaaS** — you rent the machine and its network placement, and you own the drivers, the scheduler and every idle hour. - **PaaS** — managed training and serving platforms (Vertex AI, SageMaker, Azure AI Foundry) that hide the cluster but not the reservation. - **SaaS** — finished applications where the model is an implementation detail. - **Model-as-a-service** — a hosted [API](https://howaiworks.ai/glossary/api) billed per token rather than per hour. This is the genuinely new one: it moves utilisation risk to the provider, who multiplexes your bursty traffic against everyone else's and can therefore restore the elasticity that raw accelerators lost. ## Real-World Applications **Checkpointing cadence is a cloud decision, not a training detail.** In a 54-day pre-training snapshot on 16,384 H100s, Meta logged 466 job interruptions, 419 of them unexpected — about **one every three hours**. That is not unreliable hardware: `16,384 x 54 / 419` works out to roughly 2,100 GPU-days, or **5.8 GPU-years between failures for any individual chip**. The cluster fails constantly because it is large, and doubling it halves the time between interruptions. Any rented cluster past a few thousand accelerators must therefore be architected around restart, not around uptime. **Where the data already is decides where the compute goes.** A 15-trillion-token corpus is roughly 60 TB of text at about 4 bytes per token; sustaining 10 Gb/s, copying it between providers takes `60e12 x 8 / 1e10 ≈ 48,000` seconds — **over 13 hours of pure transfer** before egress billing. Google dropped exit fees in January 2024 and AWS followed that March, and the EU Data Act bans profit-making switching charges from 12 January 2027 — but those cover a full exit, not the routine cross-provider traffic a multi-cloud pipeline generates every day. **Training and [inference](https://howaiworks.ai/glossary/inference) pull deployment in opposite directions.** Training is a synchronous burst that cares about interconnect and cheap power and can run wherever both exist. Serving is a steady, latency-sensitive load that must sit near users: light in fibre covers about 200,000 km/s, so a Frankfurt–Virginia round trip has a **66 ms floor** before any routing overhead — comparable to two or three tokens of generation time, added to every request, fixable only by moving the [deployment](https://howaiworks.ai/glossary/model-deployment) and not by buying faster chips. ## Challenges **Scale-to-zero barely works.** A 70B model in BF16 is 140 GB of weights. Loading that into accelerator memory takes about 14 seconds at 10 GB/s from local NVMe and over a minute from object storage — against a web container of perhaps 100 MB that cold-starts in well under a second. So "serverless" AI endpoints mostly keep weights resident, and you pay for idle accelerators under a name that suggests you are not. **Reservation terms outlive hardware relevance.** Accelerator generations arrive roughly every two years while commitments run one to three, so a contract signed today will spend part of its term on silicon that is a generation behind — and the [energy](https://howaiworks.ai/glossary/ai-energy-consumption) per useful token on that older hardware is worse too. **Quota, not the API, is the real ceiling.** The endpoint that creates an eight-GPU instance is identical to the one that creates a small VM; what differs is whether the request is granted. Teams plan [scale](https://howaiworks.ai/glossary/scalable-ai) against documented limits and discover the binding one is an account quota negotiated with a salesperson. ## Future Trends Capacity contracts are moving upstream toward power: the scarce input is megawatts at a substation, so buyers increasingly sign against a data center's grid connection years ahead. Serving is splitting in the opposite direction — prefill and decode disaggregated onto different hardware pools because they stress compute and memory differently (see [inference optimization](https://howaiworks.ai/glossary/inference-optimization)), with the latency-critical tail pushed toward [edge AI](https://howaiworks.ai/glossary/edge-ai). And per-token pricing keeps abstracting the accelerator away entirely, which is convenient right up to the moment a data-residency rule dictates which country the weights may sit in, and hardware becomes visible again. ## Frequently Asked Questions ### How does cloud computing support AI? Cloud computing provides the massive computational power ([GPUs](https://howaiworks.ai/glossary/gpu-computing) and [TPUs](https://howaiworks.ai/glossary/tensor-processing-unit)) required to train and serve large-scale AI models that would be too expensive for most companies to run locally. ### What are the main types of cloud services? Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). ### Why can't I just start a thousand GPUs on demand the way I start a thousand CPUs? Because classic cloud elasticity is statistical: the provider oversells capacity safely because thousands of small tenants peak at different moments. A synchronous training job has no peaks to smooth — it wants every accelerator at once, adjacent on the same fast interconnect, for weeks. There is nothing to multiplex, so the provider sells you a reservation instead of an illusion of infinite capacity. ### Is it cheaper to buy accelerators than to rent them? It depends almost entirely on sustained utilisation, not on the headline hourly rate. A reservation bills whether the hardware is busy or idle, so your effective cost per unit of work is the quoted rate divided by the fraction of time the fleet does useful work — 2.5x at 40% utilisation, 1.25x at 80%. That 2x spread is larger than any discount a provider will negotiate, which is why steady large-scale users tend to own hardware and everyone else rents. ## Related ### Related terms - [AI Data Center](https://howaiworks.ai/glossary/ai-data-center) - [AI Infrastructure](https://howaiworks.ai/glossary/ai-infrastructure) - [Distributed Computing](https://howaiworks.ai/glossary/distributed-computing) - [GPU Computing](https://howaiworks.ai/glossary/gpu-computing) - [Scalable AI](https://howaiworks.ai/glossary/scalable-ai) - [Model Deployment](https://howaiworks.ai/glossary/model-deployment) --- Source: https://howaiworks.ai/glossary/cloud-computing — HowAIWorks.ai