What One AI Query Costs: Tokens, Bytes, Joules, Cents

Introduction

Everyone quotes a number. "An AI query costs a fraction of a cent." "An AI query uses ten times the energy of a search." Almost nobody shows the arithmetic, and the arithmetic is the whole story: change one setting nobody outside the serving team ever sees, and the same answer to the same question costs four orders of magnitude more or less.

So here is one answer, costed all the way down. One concrete question, one concrete open-weights model, one concrete accelerator, and every conversion in between:

tokens  →  bytes moved  →  GPU-seconds  →  joules  →  cents

Then we move one variable at a time and watch the answer slide. From Token to Transistor explains why the machine behaves this way; this post puts a price tag on it.

The Question, the Model, the Machine

The query. A 1,000-token prompt — a question plus a system prompt, the ordinary shape of a real request — and a 500-token answer. Epoch AI treats a 500-token reply, about 400 words, as a typical response length and calls the assumption "somewhat pessimistic" (Epoch AI).

The model. Llama 3.1 70B, served in FP8 — one byte per parameter. Meta's model card gives 70,553,706,496 parameters, so the weights are 70.6 GB. Its published config gives 80 layers, 8 key/value heads and a head dimension of 128, which the KV cache page turns into 320 KiB of cache per token.

The machine. Eight NVIDIA H100 SXM. Each is 80 GB of HBM3 at 3.35 TB/s, drawing up to 700 W (NVIDIA) — so the node has 640 GB of memory, 26.8 TB/s of aggregate bandwidth, and a 5.6 kW appetite. Lambda rents it at $3.99 per GPU-hour on the 8-GPU on-demand instance (Lambda), which is $31.92 an hour for the node.

The batch. 32 concurrent requests — an unremarkable production setting.

Two conversion constants fall out of that, and they are the spine of everything below:

1 GPU-second = $3.99 / 3600 = 0.1108 cents
1 GPU-second = 700 W x 1 s   = 700 joules

Get to GPU-seconds and both the money and the physics are one multiplication away.

The Chain

Tokens to bytes

Generating a token means reading the model's entire weights out of memory — that is the memory wall, and it is why the bytes, not the arithmetic, are the unit of account. Each batched conversation also carries its own KV cache, which must be re-read on every step.

Our request runs from 1,000 tokens to 1,500, so it averages 1,250 tokens of context:

cache per sequence = 1,250 tokens x 327,680 bytes = 0.41 GB
bytes per decode step = 70.6 GB (weights) + 32 x 0.41 GB (caches) = 83.7 GB

The weights are read once and serve all 32 users. That single fact is what everything else in this post is about.

Bytes to GPU-seconds

Divide the bytes by the node's bandwidth:

time per decode step = 83.7 GB / 26,800 GB/s = 3.12 ms

That 3.12 ms is what one user experiences as time-per-output-token — 320 tokens a second, faster than anyone reads. Five hundred tokens therefore occupy the node for 500 x 3.12 ms = 1.56 seconds — but the node was serving 32 people the whole time, so our share is a thirty-second of it:

decode = 1.56 node-s / 32 x 8 GPUs = 0.39 GPU-seconds

The prompt is not free either. Prefill is a dense matrix multiplication over all 1,000 tokens at once — roughly 2 x parameters x tokens = 141 TFLOPs of work (FLOPs). The node offers ~1,979 dense FP8 TFLOPS per GPU; at a realistic 40% utilisation that is 6,333 TFLOP/s, so prefill takes 22 ms of node time — and unlike decode, nobody shares it:

prefill = 0.022 node-s x 8 GPUs = 0.18 GPU-seconds
total   = 0.39 + 0.18 = 0.57 GPU-seconds

GPU-seconds to cents

0.57 GPU-s x 0.1108 cents = 0.063 cents

Six hundredths of a cent. About one-sixteenth of a penny, to answer a question, on hardware rented at list price.

GPU-seconds to joules, and the electricity bill

0.57 GPU-s x 700 W = 398 joules = 0.111 Wh

Now convert that to money. Apply the industry-average data-centre PUE of 1.54 (Uptime Institute, 2025) and the US industrial electricity price of 8.66 cents per kWh (EIA, April 2026, Table 5.3):

0.111 Wh x 1.54 = 0.171 Wh = 0.00147 cents of electricity

Compare that with the 0.063 cents of rent. Electricity is 2.3% of the bill. The cross-check works from the other end too: an H100 at 700 W with PUE 1.54 draws 1.08 kW, which at 8.66 c/kWh is 9.3 cents of electricity per hour — against a $3.99 rental price. Same 2.3%.

That ratio deserves a moment. When you pay for AI, you are almost entirely paying for silicon, buildings and somebody's margin. The joules are a rounding error on the invoice — which says nothing about whether they are a rounding error on the grid, and AI energy consumption is where that separate argument belongs.

Two sanity checks before we start moving things. Our accelerator-only 0.111 Wh sits within about 2x of the only measured production figure anyone has published — Google's instrumented median of 0.24 Wh per Gemini text prompt, which counts host CPU, idle machines and facility overhead too (arXiv 2508.15734). And the batch-1 row below lands at 2.1 Wh, reproducing the figure the energy page derives by a different route. The chain holds.

It is still a floor. It assumes peak bandwidth is sustained, counts the GPU and nothing else, and ignores activations, attention kernels and the interconnect. Real systems land above it — see the calibration on AI energy consumption, which puts metered hardware roughly 3-4x above this kind of derivation. The ratios that follow are the durable part.

Now Move One Dial

Batch size: 41x

The biggest lever, and the one no user can see. Everything above holds; only the number of simultaneous requests changes.

From batch 1 to batch 512, the same answer gets 41x cheaper — and the user's text slows from 378 tokens per second to 96. That is the whole trade, and inference optimization derives the curve properly.

Two things worth reading off the table. First, the free lunch has an end: batching amortises the fixed 70.6 GB of weights, but each user's 0.41 GB of cache is a variable cost re-read every step. They cross at 0.41 x B = 70.6, so B ≈ 172 — and sure enough, quadrupling the batch from 128 to 512 cuts cost by only 19% while making each user wait 2.3x longer. Past that point you are buying almost nothing with your users' patience.

Second, at batch 512 the prefill — 0.18 GPU-seconds that never batch away — is 69% of the total cost. Squeeze decode hard enough and the prompt becomes the bill. This is why prompt caching exists.

Quantization: 1.4x, not 2x

Drop the weights to INT4 and they halve, from 70.6 GB to 35.3 GB (quantization works the ladder). Naively that should halve the cost. It does not:

• At batch 1: 5.51 GPU-s, 0.61c — down from 1.19c. A clean 1.96x.
• At batch 32: 0.40 GPU-s, 0.045c — down from 0.063c. Only 1.4x.

The reason is in the formula. Quantization shrinks the fixed term, and the fixed term is exactly what a large batch had already amortised into insignificance. The more efficient your batching, the less quantization buys you — which is the opposite of the intuition that optimisations stack. Its real payoff at scale is indirect: the memory it frees becomes more batch.

Context length: the cache eats the batch

Same question, but the user pastes in a 100,000-token document.

The cache per sequence is now 100,250 x 320 KiB = 32.9 GB. After the weights, the node has 569 GB of HBM left — so it fits 17 such sessions. Not 32. Not 512. Seventeen. The context window has quietly deleted the batch, and there is no setting that recovers it.

decode  = 500 steps x (70.6 + 17 x 32.9) GB / 26,800 GB/s, share of 17, x8  =  5.5 GPU-s
prefill = 2 x 70.6e9 x 100,000 FLOPs / 6,333 TFLOP/s x 8                    = 17.8 GPU-s
total   = 23.4 GPU-seconds  →  2.59 cents

41x the baseline, for the same 500-token answer. And note the shape has inverted: prefill is now 76% of the cost. Long context is a compute bill on the way in and a memory bill on the way out, and the memory half is the one that ruins the economics for everybody else on the machine.

Thinking tokens: worse than linear

Now let the model reason before it answers. DeepSeek's R1 model card is unusually candid about the token count: "in the AIME test set, the previous model used an average of 12K tokens per question, whereas the new version averages 23K tokens per question" (DeepSeek-R1-0528). Hold our 70B model fixed and let it emit the same volume of hidden chain of thought before its 500 visible tokens:

Look at the last two rows. The tokens went up 1.9x; the cost went up 2.6x. Reasoning is superlinear, because the model is not just emitting more tokens — it is dragging a KV cache that grows with every one of them, and re-reading the whole thing on every step. A newer version of the same model that thinks twice as hard costs two and a half times as much to run.

This is test-time compute with an invoice attached, and it is the reason the industry's compute demand moved from training into serving.

The Trap: "That Provider Must Be Losing Money"

Here is the mistake this post exists to prevent.

An engineer benchmarks their own deployment. They rent an H100 node, load a 70B model, send it their query, and measure. It costs them 1.19 cents — the batch-1 row, because they are one person testing. Then they look up what Together AI charges for a 70B-class Llama: $1.04 per million tokens, input and output alike (Together AI). Their 1,500 tokens would cost 0.156 cents to buy.

The provider is selling for an eighth of what it costs to make. One of two wrong conclusions follows: the provider is torching venture money, or my measurement is broken.

Neither. Set the provider's revenue equal to our cost function and solve for the batch:

0.156 cents of revenue  =  cost(B)
                    →   B ≈ 9

A provider charging $1.04 per million tokens breaks even at a batch of about nine. At batch 32 it earns a 2.5x gross margin; at batch 128, about 4.4x. Nothing about the model, the hardware or the software differs from the engineer's setup. The entire gap — a factor of forty — is a scheduler setting.

The corollary is the one people find harder to accept: self-hosting to save money almost never works. Not because your engineering is bad, but because you do not have enough simultaneous users to amortise a 70.6 GB memory read. Self-host for privacy, for control, for offline use. Do not self-host for the invoice.

And one more thing hides in that price. Our 1,000 input tokens cost 0.020 cents to prefill and bill at 0.104 cents; our 500 output tokens cost 0.043 cents to decode and bill at 0.052 cents. Per token, output is about 4x more expensive to produce — and Together charges the same for both. A flat input/output price is a very good deal for whoever wrote it, which is precisely why most frontier APIs price output well above input.

The Spread

Push every dial to a realistic extreme and cost the same class of question both ways.

Both are "one AI query." Between them is a factor of more than 10,000 in money and in joules alike — which is why any headline of the form "an AI query costs X" or "an AI query uses Y watt-hours" has silently picked a row of a table it did not show you. The AI energy consumption page makes exactly this argument for the energy side, and traces the famous "3 watt-hours" figure back to a 2009 blog post multiplied by an executive's remark about dollars.

What to Keep When the Prices Rot

Every dollar figure above is a list price at the time of writing. Lambda's $3.99, Together's $1.04, EIA's 8.66 cents — all three will be wrong within a year, and possibly within a quarter.

Keep the ratios instead. They are set by the hardware and the model, not by anyone's price list:

• Batch size is worth ~40x. Set by weights-versus-cache bytes. It will still be worth ~40x when H100s are cheap.
• The batching lunch ends where the caches outweigh the weights. A formula, not a price: cache_per_seq x B = model_bytes.
• Electricity is ~2% of the rent. Set by TDP, PUE and the price of a GPU-hour — and it stays small unless accelerators get radically cheaper without getting more efficient.
• Quantization pays least where batching pays most. Structural, and permanent.
• Reasoning is superlinear. Guaranteed by the cache, whatever the model.

Redo every calculation on this page with next year's prices and a Blackwell node. The absolute numbers will all move. The shape will not.

Conclusion

One question, one model, one machine: 0.063 cents and 398 joules. That is a real number, and it is also nearly useless on its own — because the same question, on the same model, on the same machine, costs anywhere from half a thousandth of a cent to nearly a dollar depending on four settings the person quoting the number almost never states.

So when you next see "an AI query costs X," the only useful response is a list of questions. What batch? What precision? How long was the context? How many tokens did it think for? Absent those, the figure is not a measurement. It is a choice of row.

The chain itself — tokens, bytes, GPU-seconds, joules, cents — does not rot. Learn it once and you can price any model on any accelerator, including the ones that do not exist yet.

Sources

• NVIDIA H100 Tensor Core GPU — 80 GB HBM3, 3.35 TB/s, up to 700 W.
• Lambda GPU cloud pricing — H100 SXM on-demand, $3.99/GPU-hour (8x instance), at the time of writing.
• Together AI pricing — 70B-class Llama at $1.04 per million tokens, input and output, at the time of writing.
• EIA Electric Power Monthly, Table 5.3 — average US industrial electricity price, 8.66 c/kWh, April 2026.
• Uptime Institute Global Data Center Survey 2025 — average annual PUE of 1.54.
• DeepSeek-R1-0528 model card — 12K to 23K average tokens per AIME question.
• Epoch AI: How much energy does ChatGPT use? — 500 output tokens as a typical response.
• Measuring the environmental impact of delivering AI at Google scale — measured median of 0.24 Wh per Gemini text prompt.