--- source: 'https://howaiworks.ai/blog/what-one-ai-query-costs' section: blog title: 'What One AI Query Costs: Tokens, Bytes, Joules, Cents' description: >- One question, one model, one machine — costed end to end. Then four dials that move the same answer by more than 10,000x. date: '2026-07-13' author: HowAIWorks Team tags: - AI Infrastructure - Inference - LLM - GPU - Efficiency - Inference Optimization - Quantization - Memory Bandwidth - Energy Efficiency - NVIDIA category: AI Fundamentals readingTime: 11 minutes isReviewed: false --- # What One AI Query Costs: Tokens, Bytes, Joules, Cents > One question, one model, one machine — costed end to end. Then four dials that move the same answer by more than 10,000x. ## Introduction Everyone quotes a number. "An AI query costs a fraction of a cent." "An AI query uses ten times the energy of a search." Almost nobody shows the arithmetic, and the arithmetic is the whole story: change one setting nobody outside the serving team ever sees, and the same answer to the same question costs **four orders of magnitude** more or less. So here is one answer, costed all the way down. One concrete question, one concrete open-weights model, one concrete accelerator, and every conversion in between: ``` tokens → bytes moved → GPU-seconds → joules → cents ``` Then we move one variable at a time and watch the answer slide. [From Token to Transistor](https://howaiworks.ai/blog/how-ai-runs-on-silicon) explains *why* the machine behaves this way; this post puts a price tag on it. ## The Question, the Model, the Machine **The query.** A 1,000-token prompt — a question plus a system prompt, the ordinary shape of a real request — and a 500-token answer. Epoch AI treats a 500-token reply, about 400 words, as a typical response length and calls the assumption "somewhat pessimistic" ([Epoch AI](https://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use)). **The model.** [Llama 3.1 70B](https://howaiworks.ai/models/llama), served in FP8 — one byte per [parameter](https://howaiworks.ai/glossary/parameters). Meta's model card gives 70,553,706,496 parameters, so the weights are **70.6 GB**. Its published config gives 80 layers, 8 key/value heads and a head dimension of 128, which the [KV cache](https://howaiworks.ai/glossary/kv-cache) page turns into **320 KiB of cache per token**. **The machine.** Eight NVIDIA H100 SXM. Each is 80 GB of HBM3 at **3.35 TB/s**, drawing **up to 700 W** ([NVIDIA](https://www.nvidia.com/en-us/data-center/h100/)) — so the node has **640 GB** of memory, **26.8 TB/s** of aggregate bandwidth, and a **5.6 kW** appetite. Lambda rents it at **$3.99 per GPU-hour** on the 8-GPU on-demand instance ([Lambda](https://lambda.ai/pricing)), which is **$31.92 an hour** for the node. **The batch.** 32 concurrent requests — an unremarkable production setting. Two conversion constants fall out of that, and they are the spine of everything below: ``` 1 GPU-second = $3.99 / 3600 = 0.1108 cents 1 GPU-second = 700 W x 1 s = 700 joules ``` Get to GPU-seconds and both the money and the physics are one multiplication away. ## The Chain ### Tokens to bytes Generating a token means reading the model's entire weights out of memory — that is the [memory wall](https://howaiworks.ai/glossary/memory-wall), and it is why the bytes, not the arithmetic, are the unit of account. Each batched conversation also carries its own KV cache, which must be re-read on every step. Our request runs from 1,000 tokens to 1,500, so it averages 1,250 tokens of context: ``` cache per sequence = 1,250 tokens x 327,680 bytes = 0.41 GB bytes per decode step = 70.6 GB (weights) + 32 x 0.41 GB (caches) = 83.7 GB ``` The weights are read **once** and serve all 32 users. That single fact is what everything else in this post is about. ### Bytes to GPU-seconds Divide the bytes by the node's bandwidth: ``` time per decode step = 83.7 GB / 26,800 GB/s = 3.12 ms ``` That 3.12 ms is what one user experiences as time-per-output-token — 320 tokens a second, faster than anyone reads. Five hundred tokens therefore occupy the node for `500 x 3.12 ms = 1.56 seconds` — but the node was serving 32 people the whole time, so **our** share is a thirty-second of it: ``` decode = 1.56 node-s / 32 x 8 GPUs = 0.39 GPU-seconds ``` The prompt is not free either. Prefill is a dense [matrix multiplication](https://howaiworks.ai/glossary/matrix-multiplication) over all 1,000 tokens at once — roughly `2 x parameters x tokens` = **141 TFLOPs** of work ([FLOPs](https://howaiworks.ai/glossary/flops)). The node offers ~1,979 dense FP8 TFLOPS per GPU; at a realistic 40% utilisation that is 6,333 TFLOP/s, so prefill takes 22 ms of node time — and unlike decode, nobody shares it: ``` prefill = 0.022 node-s x 8 GPUs = 0.18 GPU-seconds total = 0.39 + 0.18 = 0.57 GPU-seconds ``` ### GPU-seconds to cents ``` 0.57 GPU-s x 0.1108 cents = 0.063 cents ``` **Six hundredths of a cent.** About one-sixteenth of a penny, to answer a question, on hardware rented at list price. ### GPU-seconds to joules, and the electricity bill ``` 0.57 GPU-s x 700 W = 398 joules = 0.111 Wh ``` Now convert *that* to money. Apply the industry-average data-centre PUE of **1.54** ([Uptime Institute, 2025](https://uptimeinstitute.com/resources/research-and-reports/uptime-institute-global-data-center-survey-results-2025)) and the US industrial electricity price of **8.66 cents per kWh** (EIA, April 2026, [Table 5.3](https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_3)): ``` 0.111 Wh x 1.54 = 0.171 Wh = 0.00147 cents of electricity ``` Compare that with the 0.063 cents of rent. **Electricity is 2.3% of the bill.** The cross-check works from the other end too: an H100 at 700 W with PUE 1.54 draws 1.08 kW, which at 8.66 c/kWh is **9.3 cents of electricity per hour** — against a **$3.99** rental price. Same 2.3%. That ratio deserves a moment. When you pay for AI, you are almost entirely paying for silicon, buildings and somebody's margin. The joules are a rounding error *on the invoice* — which says nothing about whether they are a rounding error on the grid, and [AI energy consumption](https://howaiworks.ai/glossary/ai-energy-consumption) is where that separate argument belongs. **Two sanity checks before we start moving things.** Our accelerator-only 0.111 Wh sits within about 2x of the only measured production figure anyone has published — Google's instrumented median of **0.24 Wh** per Gemini text prompt, which counts host CPU, idle machines and facility overhead too ([arXiv 2508.15734](https://arxiv.org/abs/2508.15734)). And the batch-1 row below lands at 2.1 Wh, reproducing the figure the energy page derives by a different route. The chain holds. It is still a **floor**. It assumes peak bandwidth is sustained, counts the GPU and nothing else, and ignores activations, attention kernels and the interconnect. Real systems land above it — see the calibration on [AI energy consumption](https://howaiworks.ai/glossary/ai-energy-consumption), which puts metered hardware roughly 3-4x above this kind of derivation. The *ratios* that follow are the durable part. ## Now Move One Dial ### Batch size: 41x The biggest lever, and the one no user can see. Everything above holds; only the number of simultaneous requests changes. | Batch | Bytes per decode step | Time per token | GPU-s per answer | Cost per answer | |---|---|---|---|---| | 1 | 71.0 GB | 2.65 ms | 10.78 | **1.19c** | | 8 | 73.8 GB | 2.75 ms | 1.56 | 0.17c | | 32 | 83.7 GB | 3.12 ms | 0.57 | **0.063c** | | 128 | 123.0 GB | 4.59 ms | 0.32 | 0.036c | | 512 | 280.3 GB | 10.46 ms | 0.26 | **0.029c** | From batch 1 to batch 512, the same answer gets **41x cheaper** — and the user's text slows from 378 tokens per second to 96. That is the whole trade, and [inference optimization](https://howaiworks.ai/glossary/inference-optimization) derives the curve properly. Two things worth reading off the table. First, **the free lunch has an end**: batching amortises the fixed 70.6 GB of weights, but each user's 0.41 GB of cache is a variable cost re-read every step. They cross at `0.41 x B = 70.6`, so **B ≈ 172** — and sure enough, quadrupling the batch from 128 to 512 cuts cost by only 19% while making each user wait 2.3x longer. Past that point you are buying almost nothing with your users' patience. Second, at batch 512 the **prefill** — 0.18 GPU-seconds that never batch away — is 69% of the total cost. Squeeze decode hard enough and the prompt becomes the bill. This is why prompt caching exists. ### Quantization: 1.4x, not 2x Drop the weights to INT4 and they halve, from 70.6 GB to 35.3 GB ([quantization](https://howaiworks.ai/glossary/quantization) works the ladder). Naively that should halve the cost. It does not: - **At batch 1**: 5.51 GPU-s, **0.61c** — down from 1.19c. A clean **1.96x**. - **At batch 32**: 0.40 GPU-s, **0.045c** — down from 0.063c. Only **1.4x**. The reason is in the formula. Quantization shrinks the *fixed* term, and the fixed term is exactly what a large batch had already amortised into insignificance. **The more efficient your batching, the less quantization buys you** — which is the opposite of the intuition that optimisations stack. Its real payoff at scale is indirect: the memory it frees becomes more batch. ### Context length: the cache eats the batch Same question, but the user pastes in a 100,000-token document. The cache per sequence is now `100,250 x 320 KiB` = **32.9 GB**. After the weights, the node has 569 GB of HBM left — so it fits **17** such sessions. Not 32. Not 512. Seventeen. The [context window](https://howaiworks.ai/glossary/context-window) has quietly deleted the batch, and there is no setting that recovers it. ``` decode = 500 steps x (70.6 + 17 x 32.9) GB / 26,800 GB/s, share of 17, x8 = 5.5 GPU-s prefill = 2 x 70.6e9 x 100,000 FLOPs / 6,333 TFLOP/s x 8 = 17.8 GPU-s total = 23.4 GPU-seconds → 2.59 cents ``` **41x the baseline, for the same 500-token answer.** And note the shape has inverted: prefill is now **76%** of the cost. Long context is a compute bill on the way in and a memory bill on the way out, and the memory half is the one that ruins the economics for everybody else on the machine. ### Thinking tokens: worse than linear Now let the model reason before it answers. DeepSeek's [R1](https://howaiworks.ai/models/deepseek) model card is unusually candid about the token count: "in the AIME test set, the previous model used an average of **12K tokens** per question, whereas the new version averages **23K tokens** per question" ([DeepSeek-R1-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528)). Hold our 70B model fixed and let it emit the same volume of hidden [chain of thought](https://howaiworks.ai/glossary/chain-of-thought) before its 500 visible tokens: | Hidden tokens | Output tokens | Avg. cache/seq | GPU-s per answer | Cost | |---|---|---|---|---| | 0 | 500 | 0.41 GB | 0.57 | 0.063c | | 12,000 | 12,500 | 2.38 GB | 17.3 | **1.91c** | | 23,000 | 23,500 | 4.18 GB | 45.0 | **4.98c** | Look at the last two rows. The tokens went up **1.9x**; the cost went up **2.6x**. Reasoning is *superlinear*, because the model is not just emitting more tokens — it is dragging a KV cache that grows with every one of them, and re-reading the whole thing on every step. A newer version of the same model that thinks twice as hard costs two and a half times as much to run. This is [test-time compute](https://howaiworks.ai/glossary/test-time-compute) with an invoice attached, and it is the reason the industry's compute demand moved from training into serving. ## The Trap: "That Provider Must Be Losing Money" Here is the mistake this post exists to prevent. An engineer benchmarks their own deployment. They rent an H100 node, load a 70B model, send it their query, and measure. It costs them **1.19 cents** — the batch-1 row, because they are one person testing. Then they look up what Together AI charges for a 70B-class Llama: **$1.04 per million tokens**, input and output alike ([Together AI](https://www.together.ai/pricing)). Their 1,500 tokens would cost **0.156 cents** to buy. The provider is selling for **an eighth of what it costs to make.** One of two wrong conclusions follows: *the provider is torching venture money*, or *my measurement is broken*. Neither. Set the provider's revenue equal to our cost function and solve for the batch: ``` 0.156 cents of revenue = cost(B) → B ≈ 9 ``` **A provider charging $1.04 per million tokens breaks even at a batch of about nine.** At batch 32 it earns a 2.5x gross margin; at batch 128, about 4.4x. Nothing about the model, the hardware or the software differs from the engineer's setup. The entire gap — a factor of forty — is a scheduler setting. The corollary is the one people find harder to accept: **self-hosting to save money almost never works.** Not because your engineering is bad, but because you do not have enough simultaneous users to amortise a 70.6 GB memory read. Self-host for privacy, for control, for offline use. Do not self-host for the invoice. And one more thing hides in that price. Our 1,000 input tokens cost **0.020 cents** to prefill and bill at **0.104 cents**; our 500 output tokens cost **0.043 cents** to decode and bill at **0.052 cents**. Per token, output is about **4x more expensive to produce** — and Together charges the same for both. A flat input/output price is a very good deal for whoever wrote it, which is precisely why most frontier APIs price output well above input. ## The Spread Push every dial to a realistic extreme and cost the *same class of question* both ways. | | Model | Batch | Output tokens | Energy | Cost | |---|---|---|---|---|---| | **Cheapest** | Llama 3.1 8B, INT4, one H100 | 256 | 500 | 0.009 Wh | **0.0052c** | | **Dearest** | Llama 3.1 70B, FP8, 8x H100 | 1 | 23,500 | 102 Wh | **58c** | Both are "one AI query." Between them is a factor of **more than 10,000** in money and in joules alike — which is why any headline of the form *"an AI query costs X"* or *"an AI query uses Y watt-hours"* has silently picked a row of a table it did not show you. The [AI energy consumption](https://howaiworks.ai/glossary/ai-energy-consumption) page makes exactly this argument for the energy side, and traces the famous "3 watt-hours" figure back to a 2009 blog post multiplied by an executive's remark about dollars. ## What to Keep When the Prices Rot Every dollar figure above is a **list price at the time of writing**. Lambda's $3.99, Together's $1.04, EIA's 8.66 cents — all three will be wrong within a year, and possibly within a quarter. Keep the **ratios** instead. They are set by the hardware and the model, not by anyone's price list: - **Batch size is worth ~40x.** Set by weights-versus-cache bytes. It will still be worth ~40x when H100s are cheap. - **The batching lunch ends where the caches outweigh the weights.** A formula, not a price: `cache_per_seq x B = model_bytes`. - **Electricity is ~2% of the rent.** Set by TDP, PUE and the price of a GPU-hour — and it stays small unless accelerators get radically cheaper without getting more efficient. - **Quantization pays least where batching pays most.** Structural, and permanent. - **Reasoning is superlinear.** Guaranteed by the cache, whatever the model. Redo every calculation on this page with next year's prices and a Blackwell node. The absolute numbers will all move. The shape will not. ## Conclusion One question, one model, one machine: **0.063 cents and 398 joules**. That is a real number, and it is also nearly useless on its own — because the same question, on the same model, on the same machine, costs anywhere from half a thousandth of a cent to nearly a dollar depending on four settings the person quoting the number almost never states. So when you next see "an AI query costs X," the only useful response is a list of questions. *What batch? What precision? How long was the context? How many tokens did it think for?* Absent those, the figure is not a measurement. It is a choice of row. The chain itself — tokens, bytes, GPU-seconds, joules, cents — does not rot. Learn it once and you can price any model on any accelerator, including the ones that do not exist yet. ## Sources - [NVIDIA H100 Tensor Core GPU](https://www.nvidia.com/en-us/data-center/h100/) — 80 GB HBM3, 3.35 TB/s, up to 700 W. - [Lambda GPU cloud pricing](https://lambda.ai/pricing) — H100 SXM on-demand, $3.99/GPU-hour (8x instance), at the time of writing. - [Together AI pricing](https://www.together.ai/pricing) — 70B-class Llama at $1.04 per million tokens, input and output, at the time of writing. - [EIA Electric Power Monthly, Table 5.3](https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_3) — average US industrial electricity price, 8.66 c/kWh, April 2026. - [Uptime Institute Global Data Center Survey 2025](https://uptimeinstitute.com/resources/research-and-reports/uptime-institute-global-data-center-survey-results-2025) — average annual PUE of 1.54. - [DeepSeek-R1-0528 model card](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) — 12K to 23K average tokens per AIME question. - [Epoch AI: How much energy does ChatGPT use?](https://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use) — 500 output tokens as a typical response. - [Measuring the environmental impact of delivering AI at Google scale](https://arxiv.org/abs/2508.15734) — measured median of 0.24 Wh per Gemini text prompt. ## Frequently Asked Questions ### How much does one AI query cost to serve? For a 70B open-weights model on rented H100s, a 1,000-token prompt with a 500-token answer costs about 0.063 cents at a production batch of 32 — roughly one-sixteenth of a cent. But the same question costs 1.2 cents on a GPU serving one user, and nearly 60 cents if the model thinks for 23,000 hidden tokens first. The configuration is the number. ### Why is running a model on your own GPU more expensive than an API? Because you are batch size one. Generating a token means reading the model's entire weights out of memory, and a provider spreads that single read across dozens of users while you spread it across one. On rented hardware at list prices, the same answer costs about 8x more to produce alone than a provider charges to sell it to you. ### How much of an AI bill is electricity? About 2%. An H100 drawing 700 W, with data-centre overhead at the industry-average PUE of 1.54, burns roughly 9.3 cents of electricity an hour at US industrial rates — against a list rental price of $3.99 per GPU-hour. The other 97-98% is silicon, buildings and margin. ### Why does attaching a long document make a query so much more expensive? Two costs at once. Reading the document is a one-off compute charge, and the key-value cache it produces then has to be re-read from memory on every single token of the answer. Worse, that cache is so large it caps how many users fit on the machine — which collapses the batch size that made the query cheap in the first place. ### Do reasoning models cost proportionally more than normal models? They cost more than proportionally. Doubling the hidden thinking tokens from about 12,000 to 23,000 raises the cost of one answer by roughly 2.6x, not 1.9x — because the KV cache the model must re-read grows along with the chain of thought, so each extra token is also more expensive than the one before it. --- Source: https://howaiworks.ai/blog/what-one-ai-query-costs — HowAIWorks.ai