---
source: 'https://howaiworks.ai/glossary/ai-energy-consumption'
section: glossary
title: AI Energy Consumption
description: >-
  How much energy an AI query really uses, how training compares with serving,
  and how to read the numbers without believing either the hype or the panic.
tags:
  - Energy Efficiency
  - AI Infrastructure
  - Inference
  - Data Centers
  - AI Hardware
  - LLM
  - Efficiency
category: Infrastructure & Hardware
datePublished: '2026-07-13'
lastUpdated: '2026-07-24'
---

# AI Energy Consumption

> How much energy an AI query really uses, how training compares with serving, and how to read the numbers without believing either the hype or the panic.

## Definition

**"How much energy does an AI query use" has no single answer — and the figure most people have
seen, roughly 3 watt-hours or "ten times a Google search", is a 2009 blog post multiplied by an
executive's offhand remark about money.** Energy per query is not a property of AI. It is the
output of five quantities: the accelerator's power draw, its memory bandwidth, the size of the
model, how many requests the server answers at once, and how many tokens the model emits before it
stops.

Move those five dials across their ordinary ranges and the *same question, answered well* costs
anywhere from about **0.001 watt-hours to several tens of watt-hours**. Four orders of magnitude —
which is why every headline number is meaningless without its assumptions, and why what follows
derives the range from a datasheet rather than quoting anybody.

## How It Works

### The machine constant: joules per gigabyte moved

Producing a token is a memory problem, not an arithmetic one. Each token requires reading **every
weight of the model out of memory**, and does so little maths with them that the chip spends almost
the whole step waiting for bytes; the [memory wall](https://howaiworks.ai/glossary/memory-wall) page derives this and
shows an H100 at **0.34%** of its arithmetic capacity while a model "generates fast".

If the accelerator is essentially always moving bytes, its energy per token is simply *the time to
move them* times *the power drawn while moving them* — and both numbers are on the datasheet.
NVIDIA's H100 SXM — the Hopper-generation accelerator that trained most of today's open models — is
rated at **3.35 TB/s** of bandwidth and **up to 700 W**
([NVIDIA datasheet](https://www.nvidia.com/en-us/data-center/h100/), read July 2026). Divide:

`700 J/s ÷ 3,350 GB/s ≈ 0.21 joules per gigabyte moved`

Read that **0.21 J/GB as a dated input, not a constant of nature.** Every other figure on this page
is this number times a count of gigabytes, so all of them carry its generation with them — anchor to
a newer chip and they move together. What does *not* move is the mechanism the number comes from:
energy per byte is just *watts ÷ bytes-per-second*, so the energy cost of a token is a property of
the chip's **power-to-bandwidth ratio**, and a faster accelerator that draws proportionally more
power buys nothing. Bandwidth per watt, not FLOPS, decides the bill. Redo the division for a
Blackwell B200 — the generation now filling data centres, rated **up to 1,000 W** over **8 TB/s**
([NVIDIA](https://docs.nvidia.com/enterprise-reference-architectures/hgx-ai-factory/latest/components.html),
read July 2026) — and you get `0.125 J/GB`, a **1.7× improvement**. That one line is the generational
efficiency gain, and it is why the *method* here does not rot even as the input does: keep the ratio,
refresh the datasheet.

### From gigabytes to joules per token

The bytes a decode step reads are the model's weights, so:

`energy per token ≈ 0.21 J/GB × (model size in GB) ÷ (batch size)`

The batch divides because the weights are read **once** and used for every request in it — 64 users
share one trip through memory. That division is the most important fact in AI energy accounting,
and it is the one the headlines omit.

Work it for Llama 3.1 70B at 8-bit precision, one byte per weight, **70.6 GB** per step:

- Batch of 1 (one user alone on the GPU): `0.21 × 70.6 ÷ 1 = 14.8 J per token`
- Batch of 32 (an ordinary production server): `0.21 × 70.6 ÷ 32 = 0.46 J per token`

A 500-token answer — the length [Epoch AI](https://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use)
treats as a typical reply, close to the 400-token reference response in
[Mistral's life-cycle study](https://mistral.ai/news/our-contribution-to-a-global-environmental-standard-for-ai/) —
therefore costs `500 × 14.8 J = 2.05 Wh` at batch 1, and `500 × 0.46 J = 0.064 Wh` at batch 32.

Same model, same chip, same answer: **a factor of 32, bought by nothing but having other users.**
And note where the batch-1 figure lands — about 2 Wh, uncomfortably close to the famous "3 Wh per
query". That is not a coincidence, and we will come back to it.

### The accelerator is not the whole machine

That arithmetic counts the GPU alone. Real serving also burns power in host CPU and DRAM, in
networking, in **idle machines held in reserve**, and in cooling and power distribution — the
[AI data center](https://howaiworks.ai/glossary/ai-data-center) page owns that ledger.

Google is the only operator to have published both halves. Its 2025 instrumentation of Gemini gives
a comprehensive median of **0.24 Wh** per text prompt, decomposed as **0.14 Wh** accelerator,
**0.06 Wh** host CPU and DRAM, **0.02 Wh** idle machines, **0.02 Wh** data-centre overhead
([arXiv 2508.15734](https://arxiv.org/abs/2508.15734)). Its stated scope factor is precise: "a
scaling of **1.72** would need to be applied to active AI accelerator energy consumption to include
the energy consumed in a production serving environment." Treat the more widely quoted **2.4×**
(0.10 Wh vs 0.24 Wh) with care — Google says that 0.10 Wh figure also comes from "considering a
sample of *more highly utilized machines*", so part of the gap is boundary and part of it is
choosing your best data centres.

### Now check the derivation against a real meter — and mind the gap

The [ML.Energy leaderboard](https://ml.energy/leaderboard), built on the peer-reviewed
[ML.Energy benchmark](https://arxiv.org/abs/2505.06371) (NeurIPS 2025, Datasets & Benchmarks), reads
GPU energy counters under production vLLM serving. Its published data for Llama 3.1 70B on H100
meters **3.76 J per token at batch 8**, **0.48 at batch 256** and **0.37 at batch 1,024**. (The
per-token figures are in the leaderboard's data, not in the paper, which reports joules per
generation — cite accordingly.)

Put our formula next to that and the honest verdict is: **right shape, optimistic magnitude.** The
division predicts 0.46 J/token at a batch of 32; the machine does not reach that figure until a
batch of about **256**. The derivation is a *floor* — roughly three to four times below what silicon
actually delivers — and the three reasons it is low are worth more than the number itself:

- **It counts only the weights.** Every batched conversation also has a
  [KV cache](https://howaiworks.ai/glossary/kv-cache) that must be read on every step. At a batch of 32 those caches add
  more than half again to the bytes moved, and past a batch of about fifty they overtake the weights
  entirely — which is why energy per token stops falling with batch size long before the formula says
  it should.
- **A 70B model is not served on one GPU.** ML.Energy shards it across four H100s, and all four draw
  power while each moves a quarter of the weights. The datasheet division silently assumes one chip.
- **No real system sustains peak bandwidth**, and rated power is not measured power. ML.Energy's own
  headline methodological finding is that estimating from a chip's TDP — exactly what we just did —
  can **overestimate** energy by up to 4.1× on some workloads, even as it underestimates here.

Keep the formula for what it is good at: it tells you *what depends on what*, it does not rot, and
you can redo it for any chip and any model in one line. For a number to quote, use a meter.

### Two teams measured the same thing and got 37× apart

This is the most important fact on the page, and it is not modelled — it is two sets of instrumented
readings.

ML.Energy meters Llama-3-70B on H100 under vLLM with continuous batching at roughly **0.046 Wh per
request**. The [AI Energy Score](https://huggingface.co/spaces/AIEnergyScore/Leaderboard) — also
measured, also on H100, also counting only the GPU — reports the same size of model at **1.72 Wh per
request**. That is a **37-fold** disagreement between two careful teams, and neither is wrong. One
harness runs a production server; the other runs plain PyTorch at **batch size one**, where most of
the GPU-seconds are spent idle but powered.

The serving configuration *is* the number. Anyone who quotes you watt-hours per query without
telling you the batch size, the engine and the hardware has not told you anything at all — and that
is true even when they measured it.

### What breaks: the same question, three ways

Push the dials to their realistic extremes. All accelerator-only, same H100, same dated 0.21 J/GB
input — and remember from the calibration above that these are **floors**, not forecasts. What the table is for
is the *spread*, which is the point no headline survives:

| Configuration | Bytes/token | Batch | Tokens | Energy |
|---|---|---|---|---|
| 8B model, 4-bit, high batch | 4.0 GB | 64 | 300 | **0.0011 Wh** |
| 70B model, FP8, production batch | 70.6 GB | 32 | 500 | **0.064 Wh** |
| 70B reasoning model, FP8, one user | 70.6 GB | 1 | 9,500 | **39 Wh** |

The last row is not a strawman, and its token count is measured rather than assumed: DeepSeek
reports R1 averaging **8,793 thinking tokens** per competition maths problem, against **711 output
tokens** for GPT-4o on *the same problems*
([arXiv 2501.12948v2](https://arxiv.org/abs/2501.12948v2), the Nature version — the figures are not in the v1 preprint). OpenAI says the same of its own models —
they "may generate anywhere from a few hundred to **tens of thousands of reasoning tokens**", billed
as output tokens, with **25,000 tokens** of recommended headroom
([OpenAI](https://developers.openai.com/api/docs/guides/reasoning)). A long
[chain of thought](https://howaiworks.ai/glossary/chain-of-thought) nobody reads is still every weight in the model,
pulled from memory, once per hidden token.

Between the top row and the bottom is a factor of about **35,000**. Both are "one AI query". Any
headline giving a single watt-hour figure has silently picked a row. Microsoft's modelling in
*Joule* reaches the same place by another route: a median frontier query at ~300 output tokens costs
**0.31 Wh**, while the same models in long-reasoning mode at ~5,000 tokens cost **3.91 Wh** — a
**13× jump** from nothing but letting the model think
([arXiv 2509.20241](https://arxiv.org/abs/2509.20241)).

Two caveats, which shrink the numbers but not the ratio: holding the chip at a full 700 W overstates
the low-batch rows (Epoch AI assumes a fleet runs at ~**70% of peak power**), and these are
bandwidth *ceilings*, which real systems fall below once the [KV cache](https://howaiworks.ai/glossary/kv-cache) takes
its share of the bus.

## Real-World Applications

**Training is enormous and happens once; serving is small and happens forever.** Meta's model card
reports Llama 3.1 405B consumed **30.84 million GPU-hours** on "H100-80GB (TDP of 700W)" hardware
([model card](https://huggingface.co/meta-llama/Llama-3.1-405B)). Meta publishes hours, not energy,
so every "Llama used X GWh" figure in circulation is somebody's multiplication. Here is ours, using
Meta's own TDP: `30.84 × 10⁶ h × 0.7 kW = 21.6 GWh` at the GPUs. (Dividing Meta's reported **8,930
tons of CO₂e — the *location-based* figure**; its market-based figure is zero, because Meta matches
its electricity with renewables — by that energy gives ~414 g/kWh, an ordinary grid average, which
suggests the carbon figure also counts silicon only, with no facility overhead in it.)

Now amortise. Serving the same 405B at batch 32 costs `0.21 × 405 ÷ 32 = 2.65 J` per token, so a
500-token answer is **0.37 Wh**. Therefore:

`21.6 × 10⁹ Wh ÷ 0.37 Wh ≈ 59 billion answers`

Fifty-nine billion — about seven for every person alive. OpenAI's own usage paper reports ChatGPT
users sending **18 billion messages a week** by July 2025
([Chatterji et al.](https://cdn.openai.com/pdf/a253471f-8260-40c6-a2cc-aa93fe9f142e/economic-research-chatgpt-usage-paper.pdf)).
At that rate the largest open training run of its generation is repaid in **about three weeks of one
product's traffic**, after which every joule the model spends is a serving joule. Google said as
much in 2022: "ML training and inference are only 10%–15% of Google's total energy use for each of
the last three years, each year split ⅗ for inference and ⅖ for training"
([Google Research](https://research.google/blog/good-news-about-the-carbon-footprint-of-machine-learning-training/)).
The public argument is still about training. The energy is not. The honest qualifier: the crossover
is a question about **traffic, not physics** — a model with a thousandth of ChatGPT's usage never
reaches 59 billion answers, and its training run dominates for life.

**Where the number everyone quotes actually came from.** The chain is short and every link checkable.

1. **January 2009** — Google's Urs Hölzle writes that an average search "amounts to 0.0003 kWh of
   energy per search, or 1 kJ", that is **0.3 Wh**
   ([Google blog](https://googleblog.blogspot.com/2009/01/powering-google-search.html)). Google has
   never updated it. It is still the search side of every AI comparison published today.
2. **February 2023** — Reuters reports that Alphabet chairman John Hennessy said an exchange with a
   large language model likely cost ten times more than a standard keyword search
   ([Reuters](https://www.reuters.com/technology/tech-giants-ai-like-bing-bard-poses-billion-dollar-search-problem-2023-02-22/)).
   That is the paper's own indirect speech, not a quotation — and he is talking about **dollars**, in
   a story about a billion-dollar expense line, whose thesis is that "it's inference costs you have
   to drive down". Not watt-hours.
3. **October 2023** — a *Joule* commentary by Alex de Vries combines the two: "As a standard Google
   search reportedly uses 0.3 Wh of electricity, this suggests an electricity consumption of
   approximately 3 Wh per LLM interaction." He then reaches for a second source and divides:
   SemiAnalysis — a research firm publishing a **cost** model, not an energy measurement — had
   estimated that ChatGPT needed 3,617 rated servers and answered 195 million requests a day, from
   which de Vries derives 564 MWh a day and therefore "**at most, 2.9 Wh per request**"
   ([Joule 7, 2191–2194](https://doi.org/10.1016/j.joule.2023.09.004)). Both inputs are
   SemiAnalysis's estimates; the division is his. *At most*: an upper bound, and he said so.
4. **January 2024** — the IEA's *Electricity 2024* writes, in a single sentence of body text, of
   "the **average** electricity demand of a typical Google search (0.3 Wh of electricity)" compared
   "to OpenAI's ChatGPT (**2.9 Wh per request**)" ([p.34](https://www.iea.org/reports/electricity-2024)).
   It does not merely drop the words "at most" — it **relabels the upper bound as an average**. A
   ceiling became a central estimate under an intergovernmental letterhead, and the world has cited
   the IEA ever since.

Note the circularity, too. De Vries obtained the ~3 Wh figure *by multiplying the 0.3 Wh search
number* by the chairman's "ten times". The IEA then sets 0.3 Wh against 2.9 Wh as though the
roughly tenfold ratio were a finding. It is an input.

And neither number was ever measured, and they do not even share a boundary: Google's 2009 figure
explicitly includes "other work performed before your search even starts (such as building the
search index)" — the search equivalent of *training* — which every AI per-query figure excludes.

**Be equally hard on the vendors.** Sam Altman's June 2025 essay asserts "the average query uses
about 0.34 watt-hours"
([The Gentle Singularity](https://blog.samaltman.com/the-gentle-singularity)) — no methodology, no
boundary, no model named, nothing published. It is unverifiable as written, and it is a flattering
number. Mistral went much further, commissioning a full life-cycle assessment of Mistral Large 2
covering training plus eighteen months of serving: **20.4 ktCO₂e**, **281,000 m³** of water, **1.14
gCO₂e per 400-token response** — and **no watt-hour figure at all**. The most thorough environmental
disclosure any lab has published does not contain the quantity this page is about.

**Choosing how to serve a model is choosing its energy bill.** The levers are already in the table.
[Quantization](https://howaiworks.ai/glossary/quantization) halves the bytes and roughly halves the joules;
[Mixture-of-Experts](https://howaiworks.ai/glossary/mixture-of-experts) reads only active parameters (DeepSeek-V3
activates 37B of 671B); batching divides; [Edge AI](https://howaiworks.ai/glossary/edge-ai) removes the data centre
entirely for queries a small model can handle. None of these is an environmental initiative — they
are the optimisations a team does to cut its cloud bill, which is why per-query energy falls so
fast. Google measured a **33× drop** in energy per median prompt in the twelve months to May 2025.

## Key Concepts

- **Energy per token, not energy per query.** A query has no fixed size. The unit that behaves
  predictably is the token, and the rule of thumb is `0.21 J/GB × model GB ÷ batch`, where the
  0.21 is one dated chip's power-to-bandwidth ratio (an H100's) — a floor, not a forecast. Reasoning length, context and retries are all just token counts
  multiplying that.
- **The serving configuration *is* the number.** Two teams metered the same 70B model on the same
  H100, both counting only the GPU, and came out **37× apart** — one under production batching, one
  at batch size 1. "How was it served?" moves the answer further than "which model was it?", and it
  does so even between honest measurements.
- **Measured beats modelled, and almost nothing is measured.** One figure here was instrumented in
  production (Google's 0.24 Wh); one was metered in a real deployment — BLOOM's API drew **914 kWh**
  serving **230,768 requests** over eighteen days, which divides to **3.96 Wh each** (the division is
  de Vries's, not the authors'). That is a *high* number, and the authors explain why: the endpoint
  answered requests "without any batching", and they estimate roughly three-quarters of the energy
  went on merely holding the model in memory between requests — their word is "approximately", and
  they say plainly they could not measure the idle/active split
  ([Luccioni et al.](https://arxiv.org/abs/2211.02001)). Every alarming figure in wide circulation is
  a model, not a measurement.
- **Falling per-query energy does not mean falling total energy.** Per-query cost has dropped by
  more than an order of magnitude in two years while data-centre electricity rose: the IEA puts it
  at **415 TWh in 2024**, about **1.5%** of world consumption, heading for roughly **945 TWh by
  2030** ([Energy and AI](https://www.iea.org/reports/energy-and-ai)). Note the fine print — the
  IEA's "almost half" refers to half of the *net increase*, not half the total.

## Challenges

**The comparison everyone wants does not exist.** "Is an AI answer worse than a search?" needs a
current, comparably-scoped figure for both, and Google has published a 2025 number for Gemini and a
2009 number for Search. Nobody outside the hyperscalers can close that gap, and the hyperscalers
have not.

**The median hides the tail, by construction.** Google's 0.24 Wh is not the median over prompts; it
is the *average* energy of the *model* serving the 50th-percentile prompt. Google is candid that the
real distribution is skewed by "small subsets of prompts... with high token counts, which consume a
disproportionate amount of energy" — and then reports a statistic that excludes them. No mean, no
percentiles, no tail magnitude are published by anyone. As assistants become agents emitting a
hundred thousand tokens across a dozen tool calls, that unreported tail is where the energy goes.

**Almost nobody discloses at all.** Exactly two operators have published a per-query energy figure
with a stated methodology: Google (measured) and Microsoft (modelled). OpenAI has published a bare
number on a personal blog. Meta, Amazon, Anthropic and DeepSeek have published nothing. Per-query
energy also needs a query count, and those appear sporadically and unaudited — so every top-down
estimate, de Vries's included, divides an estimated total by an estimated count and inherits both
errors.

**Electricity is one ledger of several.** Google reports 0.26 mL of water per prompt alongside its
0.24 Wh, and the silicon carries a manufacturing footprint that no per-query electricity figure
captures — [semiconductor manufacturing](https://howaiworks.ai/glossary/semiconductor-manufacturing) is where that cost
lives. Presenting any single number as *the* environmental cost of AI is a category error, whichever
number you pick.

## Future Trends

The per-token trend is downward and will stay downward, because it is driven by cost rather than
conscience: each accelerator generation improves bandwidth per watt, each serving stack grows the
batch, each quantization step halves the bytes.

The per-*task* trend is upward, and it is winning. Reasoning models spend tokens to buy accuracy,
agents spend tokens to buy autonomy, and long context spends bandwidth on a
[KV cache](https://howaiworks.ai/glossary/kv-cache) that grows every turn. Each is a deliberate trade of energy for
capability, and each multiplies the token count that the falling per-token cost applies to.

So the interesting question is shifting. It is no longer "how many watt-hours is a prompt" — that
number is small, measurable and shrinking. It is how many tokens we are willing to spend on a
question, and who gets to decide. That is a product decision, not a hardware one, and it is on
nobody's datasheet.

## Frequently Asked Questions

### How much energy does one AI query actually use?

There is no single figure, and any page that gives you one is hiding its assumptions. Google measured the median Gemini text prompt at 0.24 watt-hours in 2025 — about nine seconds of a television. But the same question put to a frontier reasoning model serving one user at a time can cost a hundred times more, and a small quantized model at a large batch can cost a hundred times less.

### Is it true that an AI query uses ten times the energy of a Google search?

No, and the claim has a traceable origin. Alphabet's chairman told Reuters in February 2023 that an LLM exchange likely cost ten times more than a keyword search — he was talking about dollars, not watt-hours. A 2023 Joule commentary multiplied that ratio by Google's 2009 estimate of 0.3 Wh per search to get roughly 3 Wh per AI query, and the number has been repeated ever since. The only measured production figure, Google's own, is 0.24 Wh.

### Does training or serving a model use more energy?

Serving, once a model is popular. Training Llama 3.1 405B took Meta 30.84 million H100-hours, about 21.6 GWh at the GPUs. Serving that model at a realistic batch size, roughly 59 billion answers of 500 tokens each would consume the same amount — about three weeks of traffic for a major assistant. Google reported in 2022 that inference was about three-fifths of its machine-learning energy across 2019-2021.

### Why do published estimates of AI energy use vary by so much?

Because energy per token is roughly model size divided by batch size, and both vary by orders of magnitude between deployments. A server answering 64 users at once reads the model's weights from memory once and answers all 64, so each user's share is a 64th of what a single-user setup costs. Add reasoning tokens, quantization and hardware generation, and a four-order-of-magnitude spread is entirely ordinary.

### Do reasoning models use more energy than normal models?

Considerably more, because energy scales with tokens generated and a reasoning model generates tokens the user never sees. DeepSeek reported R1 averaging 8,793 thinking tokens per competition maths problem against 711 output tokens for GPT-4o on the same problems. Microsoft's modelling puts a long-reasoning query at 3.91 Wh against 0.31 Wh for a normal one.

### Is AI's electricity use a problem at the grid level?

That is a different question from the per-query one, and the per-query number cannot answer it. The IEA put data-centre electricity at about 415 TWh in 2024, roughly 1.5% of world consumption, and projects around 945 TWh by 2030. Even very efficient queries add up when there are billions a day, and the binding constraint is usually local grid capacity rather than the joules inside any one answer.

## Related

### Related terms

- [AI Data Center](https://howaiworks.ai/glossary/ai-data-center)
- [Inference](https://howaiworks.ai/glossary/inference)
- [Memory Wall](https://howaiworks.ai/glossary/memory-wall)
- [Distributed Training](https://howaiworks.ai/glossary/distributed-training)
- [Edge AI](https://howaiworks.ai/glossary/edge-ai)
- [Quantization](https://howaiworks.ai/glossary/quantization)

---

Source: https://howaiworks.ai/glossary/ai-energy-consumption — HowAIWorks.ai