AI Energy Consumption

Definition

"How much energy does an AI query use" has no single answer — and the figure most people have seen, roughly 3 watt-hours or "ten times a Google search", is a 2009 blog post multiplied by an executive's offhand remark about money. Energy per query is not a property of AI. It is the output of five quantities: the accelerator's power draw, its memory bandwidth, the size of the model, how many requests the server answers at once, and how many tokens the model emits before it stops.

Move those five dials across their ordinary ranges and the same question, answered well costs anywhere from about 0.001 watt-hours to several tens of watt-hours. Four orders of magnitude — which is why every headline number is meaningless without its assumptions, and why what follows derives the range from a datasheet rather than quoting anybody.

How It Works

The machine constant: joules per gigabyte moved

Producing a token is a memory problem, not an arithmetic one. Each token requires reading every weight of the model out of memory, and does so little maths with them that the chip spends almost the whole step waiting for bytes; the memory wall page derives this and shows an H100 at 0.34% of its arithmetic capacity while a model "generates fast".

If the accelerator is essentially always moving bytes, its energy per token is simply the time to move them times the power drawn while moving them — and both numbers are on the datasheet. NVIDIA's H100 SXM is rated at 3.35 TB/s of bandwidth and up to 700 W (NVIDIA). Divide:

700 J/s ÷ 3,350 GB/s ≈ 0.21 joules per gigabyte moved

Every other figure on this page is that constant times a count of gigabytes. It implies something useful: the energy cost of a token is a property of the chip's power-to-bandwidth ratio, so a faster accelerator that draws proportionally more power buys nothing. Bandwidth per watt, not FLOPS, decides the bill. Redo the division for a Blackwell B200 — up to 1,000 W, about 8 TB/s (NVIDIA) — and you get 0.125 J/GB, a 1.7× improvement. That is the generational efficiency gain, in one line, and it is why this derivation does not rot.

From gigabytes to joules per token

The bytes a decode step reads are the model's weights, so:

energy per token ≈ 0.21 J/GB × (model size in GB) ÷ (batch size)

The batch divides because the weights are read once and used for every request in it — 64 users share one trip through memory. That division is the most important fact in AI energy accounting, and it is the one the headlines omit.

Work it for Llama 3.1 70B at 8-bit precision, one byte per weight, 70.6 GB per step:

• Batch of 1 (one user alone on the GPU): 0.21 × 70.6 ÷ 1 = 14.8 J per token
• Batch of 32 (an ordinary production server): 0.21 × 70.6 ÷ 32 = 0.46 J per token

A 500-token answer — the length Epoch AI treats as a typical reply, close to the 400-token reference response in Mistral's life-cycle study — therefore costs 500 × 14.8 J = 2.05 Wh at batch 1, and 500 × 0.46 J = 0.064 Wh at batch 32.

Same model, same chip, same answer: a factor of 32, bought by nothing but having other users. And note where the batch-1 figure lands — about 2 Wh, uncomfortably close to the famous "3 Wh per query". That is not a coincidence, and we will come back to it.

The accelerator is not the whole machine

That arithmetic counts the GPU alone. Real serving also burns power in host CPU and DRAM, in networking, in idle machines held in reserve, and in cooling and power distribution — the AI data center page owns that ledger.

Google is the only operator to have published both halves. Its 2025 instrumentation of Gemini gives a comprehensive median of 0.24 Wh per text prompt, decomposed as 0.14 Wh accelerator, 0.06 Wh host CPU and DRAM, 0.02 Wh idle machines, 0.02 Wh data-centre overhead (arXiv 2508.15734). Its stated scope factor is precise: "a scaling of 1.72 would need to be applied to active AI accelerator energy consumption to include the energy consumed in a production serving environment." Treat the more widely quoted 2.4× (0.10 Wh vs 0.24 Wh) with care — Google says that 0.10 Wh figure also comes from "considering a sample of more highly utilized machines", so part of the gap is boundary and part of it is choosing your best data centres.

Now check the derivation against a real meter — and mind the gap

The ML.Energy leaderboard, built on the peer-reviewed ML.Energy benchmark (NeurIPS 2025, Datasets & Benchmarks), reads GPU energy counters under production vLLM serving. Its published data for Llama 3.1 70B on H100 meters 3.76 J per token at batch 8, 0.48 at batch 256 and 0.37 at batch 1,024. (The per-token figures are in the leaderboard's data, not in the paper, which reports joules per generation — cite accordingly.)

Put our formula next to that and the honest verdict is: right shape, optimistic magnitude. The division predicts 0.46 J/token at a batch of 32; the machine does not reach that figure until a batch of about 256. The derivation is a floor — roughly three to four times below what silicon actually delivers — and the three reasons it is low are worth more than the number itself:

• It counts only the weights. Every batched conversation also has a KV cache that must be read on every step. At a batch of 32 those caches add more than half again to the bytes moved, and past a batch of about fifty they overtake the weights entirely — which is why energy per token stops falling with batch size long before the formula says it should.
• A 70B model is not served on one GPU. ML.Energy shards it across four H100s, and all four draw power while each moves a quarter of the weights. The datasheet division silently assumes one chip.
• No real system sustains peak bandwidth, and rated power is not measured power. ML.Energy's own headline methodological finding is that estimating from a chip's TDP — exactly what we just did — can overestimate energy by up to 4.1× on some workloads, even as it underestimates here.

Keep the formula for what it is good at: it tells you what depends on what, it does not rot, and you can redo it for any chip and any model in one line. For a number to quote, use a meter.

Two teams measured the same thing and got 37× apart

This is the most important fact on the page, and it is not modelled — it is two sets of instrumented readings.

ML.Energy meters Llama-3-70B on H100 under vLLM with continuous batching at roughly 0.046 Wh per request. The AI Energy Score — also measured, also on H100, also counting only the GPU — reports the same size of model at 1.72 Wh per request. That is a 37-fold disagreement between two careful teams, and neither is wrong. One harness runs a production server; the other runs plain PyTorch at batch size one, where most of the GPU-seconds are spent idle but powered.

The serving configuration is the number. Anyone who quotes you watt-hours per query without telling you the batch size, the engine and the hardware has not told you anything at all — and that is true even when they measured it.

What breaks: the same question, three ways

Push the dials to their realistic extremes. All accelerator-only, same H100, same constant — and remember from the calibration above that these are floors, not forecasts. What the table is for is the spread, which is the point no headline survives:

The last row is not a strawman, and its token count is measured rather than assumed: DeepSeek reports R1 averaging 8,793 thinking tokens per competition maths problem, against 711 output tokens for GPT-4o on the same problems (arXiv 2501.12948v2, the Nature version — the figures are not in the v1 preprint). OpenAI says the same of its own models — they "may generate anywhere from a few hundred to tens of thousands of reasoning tokens", billed as output tokens, with 25,000 tokens of recommended headroom (OpenAI). A long chain of thought nobody reads is still every weight in the model, pulled from memory, once per hidden token.

Between the top row and the bottom is a factor of about 35,000. Both are "one AI query". Any headline giving a single watt-hour figure has silently picked a row. Microsoft's modelling in Joule reaches the same place by another route: a median frontier query at ~300 output tokens costs 0.31 Wh, while the same models in long-reasoning mode at ~5,000 tokens cost 3.91 Wh — a 13× jump from nothing but letting the model think (arXiv 2509.20241).

Two caveats, which shrink the numbers but not the ratio: holding the chip at a full 700 W overstates the low-batch rows (Epoch AI assumes a fleet runs at ~70% of peak power), and these are bandwidth ceilings, which real systems fall below once the KV cache takes its share of the bus.

Real-World Applications

Training is enormous and happens once; serving is small and happens forever. Meta's model card reports Llama 3.1 405B consumed 30.84 million GPU-hours on "H100-80GB (TDP of 700W)" hardware (model card). Meta publishes hours, not energy, so every "Llama used X GWh" figure in circulation is somebody's multiplication. Here is ours, using Meta's own TDP: 30.84 × 10⁶ h × 0.7 kW = 21.6 GWh at the GPUs. (Dividing Meta's reported 8,930 tons of CO₂e — the location-based figure; its market-based figure is zero, because Meta matches its electricity with renewables — by that energy gives ~414 g/kWh, an ordinary grid average, which suggests the carbon figure also counts silicon only, with no facility overhead in it.)

Now amortise. Serving the same 405B at batch 32 costs 0.21 × 405 ÷ 32 = 2.65 J per token, so a 500-token answer is 0.37 Wh. Therefore:

21.6 × 10⁹ Wh ÷ 0.37 Wh ≈ 59 billion answers

Fifty-nine billion — about seven for every person alive. OpenAI's own usage paper reports ChatGPT users sending 18 billion messages a week by July 2025 (Chatterji et al.). At that rate the largest open training run of its generation is repaid in about three weeks of one product's traffic, after which every joule the model spends is a serving joule. Google said as much in 2022: "ML training and inference are only 10%–15% of Google's total energy use for each of the last three years, each year split ⅗ for inference and ⅖ for training" (Google Research). The public argument is still about training. The energy is not. The honest qualifier: the crossover is a question about traffic, not physics — a model with a thousandth of ChatGPT's usage never reaches 59 billion answers, and its training run dominates for life.

Where the number everyone quotes actually came from. The chain is short and every link checkable.

1. January 2009 — Google's Urs Hölzle writes that an average search "amounts to 0.0003 kWh of energy per search, or 1 kJ", that is 0.3 Wh (Google blog). Google has never updated it. It is still the search side of every AI comparison published today.
2. February 2023 — Reuters reports that Alphabet chairman John Hennessy said an exchange with a large language model likely cost ten times more than a standard keyword search (Reuters). That is the paper's own indirect speech, not a quotation — and he is talking about dollars, in a story about a billion-dollar expense line, whose thesis is that "it's inference costs you have to drive down". Not watt-hours.
3. October 2023 — a Joule commentary by Alex de Vries combines the two: "As a standard Google search reportedly uses 0.3 Wh of electricity, this suggests an electricity consumption of approximately 3 Wh per LLM interaction." He then reaches for a second source and divides: SemiAnalysis — a research firm publishing a cost model, not an energy measurement — had estimated that ChatGPT needed 3,617 rated servers and answered 195 million requests a day, from which de Vries derives 564 MWh a day and therefore "at most, 2.9 Wh per request" (Joule 7, 2191–2194). Both inputs are SemiAnalysis's estimates; the division is his. At most: an upper bound, and he said so.
4. January 2024 — the IEA's Electricity 2024 writes, in a single sentence of body text, of "the average electricity demand of a typical Google search (0.3 Wh of electricity)" compared "to OpenAI's ChatGPT (2.9 Wh per request)" (p.34). It does not merely drop the words "at most" — it relabels the upper bound as an average. A ceiling became a central estimate under an intergovernmental letterhead, and the world has cited the IEA ever since.

Note the circularity, too. De Vries obtained the ~3 Wh figure by multiplying the 0.3 Wh search number by the chairman's "ten times". The IEA then sets 0.3 Wh against 2.9 Wh as though the roughly tenfold ratio were a finding. It is an input.

And neither number was ever measured, and they do not even share a boundary: Google's 2009 figure explicitly includes "other work performed before your search even starts (such as building the search index)" — the search equivalent of training — which every AI per-query figure excludes.

Be equally hard on the vendors. Sam Altman's June 2025 essay asserts "the average query uses about 0.34 watt-hours" (The Gentle Singularity) — no methodology, no boundary, no model named, nothing published. It is unverifiable as written, and it is a flattering number. Mistral went much further, commissioning a full life-cycle assessment of Mistral Large 2 covering training plus eighteen months of serving: 20.4 ktCO₂e, 281,000 m³ of water, 1.14 gCO₂e per 400-token response — and no watt-hour figure at all. The most thorough environmental disclosure any lab has published does not contain the quantity this page is about.

Choosing how to serve a model is choosing its energy bill. The levers are already in the table. Quantization halves the bytes and roughly halves the joules; Mixture-of-Experts reads only active parameters (DeepSeek-V3 activates 37B of 671B); batching divides; Edge AI removes the data centre entirely for queries a small model can handle. None of these is an environmental initiative — they are the optimisations a team does to cut its cloud bill, which is why per-query energy falls so fast. Google measured a 33× drop in energy per median prompt in the twelve months to May 2025.

Key Concepts

• Energy per token, not energy per query. A query has no fixed size. The unit that behaves predictably is the token, and the rule of thumb is 0.21 J/GB × model GB ÷ batch on an H100 — a floor, not a forecast. Reasoning length, context and retries are all just token counts multiplying that.
• The serving configuration is the number. Two teams metered the same 70B model on the same H100, both counting only the GPU, and came out 37× apart — one under production batching, one at batch size 1. "How was it served?" moves the answer further than "which model was it?", and it does so even between honest measurements.
• Measured beats modelled, and almost nothing is measured. One figure here was instrumented in production (Google's 0.24 Wh); one was metered in a real deployment — BLOOM's API drew 914 kWh serving 230,768 requests over eighteen days, which divides to 3.96 Wh each (the division is de Vries's, not the authors'). That is a high number, and the authors explain why: the endpoint answered requests "without any batching", and they estimate roughly three-quarters of the energy went on merely holding the model in memory between requests — their word is "approximately", and they say plainly they could not measure the idle/active split (Luccioni et al.). Every alarming figure in wide circulation is a model, not a measurement.
• Falling per-query energy does not mean falling total energy. Per-query cost has dropped by more than an order of magnitude in two years while data-centre electricity rose: the IEA puts it at 415 TWh in 2024, about 1.5% of world consumption, heading for roughly 945 TWh by 2030 (Energy and AI). Note the fine print — the IEA's "almost half" refers to half of the net increase, not half the total.

Challenges

The comparison everyone wants does not exist. "Is an AI answer worse than a search?" needs a current, comparably-scoped figure for both, and Google has published a 2025 number for Gemini and a 2009 number for Search. Nobody outside the hyperscalers can close that gap, and the hyperscalers have not.

The median hides the tail, by construction. Google's 0.24 Wh is not the median over prompts; it is the average energy of the model serving the 50th-percentile prompt. Google is candid that the real distribution is skewed by "small subsets of prompts... with high token counts, which consume a disproportionate amount of energy" — and then reports a statistic that excludes them. No mean, no percentiles, no tail magnitude are published by anyone. As assistants become agents emitting a hundred thousand tokens across a dozen tool calls, that unreported tail is where the energy goes.

Almost nobody discloses at all. Exactly two operators have published a per-query energy figure with a stated methodology: Google (measured) and Microsoft (modelled). OpenAI has published a bare number on a personal blog. Meta, Amazon, Anthropic and DeepSeek have published nothing. Per-query energy also needs a query count, and those appear sporadically and unaudited — so every top-down estimate, de Vries's included, divides an estimated total by an estimated count and inherits both errors.

Electricity is one ledger of several. Google reports 0.26 mL of water per prompt alongside its 0.24 Wh, and the silicon carries a manufacturing footprint that no per-query electricity figure captures — semiconductor manufacturing is where that cost lives. Presenting any single number as the environmental cost of AI is a category error, whichever number you pick.

Future Trends

The per-token trend is downward and will stay downward, because it is driven by cost rather than conscience: each accelerator generation improves bandwidth per watt, each serving stack grows the batch, each quantization step halves the bytes.

The per-task trend is upward, and it is winning. Reasoning models spend tokens to buy accuracy, agents spend tokens to buy autonomy, and long context spends bandwidth on a KV cache that grows every turn. Each is a deliberate trade of energy for capability, and each multiplies the token count that the falling per-token cost applies to.

So the interesting question is shifting. It is no longer "how many watt-hours is a prompt" — that number is small, measurable and shrinking. It is how many tokens we are willing to spend on a question, and who gets to decide. That is a product decision, not a hardware one, and it is on nobody's datasheet.