Definition
Performance in AI is not a single number. It is a frontier — a trade-off surface across three measurable axes, and pushing any one of them usually costs you another. Those axes are quality (how good the outputs are: accuracy on a task, or a benchmark score), speed (how fast answers arrive: latency and throughput), and cost (what each answer consumes: dollars, memory, and energy). A model that tops a leaderboard can be too slow or too expensive to ship; a model fast and cheap enough for every request may not be accurate enough to trust. So "good performance" only means something once you say on which axis, and at what expense to the others. This is what makes the word slippery: the same inference system can be called high-performance or low-performance depending entirely on which corner of the frontier you care about.
How It Works
The three axes, measured
Quality is how well the model does the job. For classification that is accuracy, precision, recall, or F1; for a large language model it is benchmark scores and human preference ratings. A single quality number — a leaderboard rank, or an intelligence-index score — is only comparable within one version of the test that produced it, a trap covered under Challenges.
Speed is two different things that people constantly conflate. Latency is one user's wait: the time to first token (TTFT), set mostly by the prefill pass that reads the whole prompt, plus the inter-token latency, the gap between successive tokens as the model decodes. If a system streams 100 tokens per second, each token is 10 ms, and a 500-token answer takes about 5 seconds to finish after the first word appears. Throughput is the whole fleet's rate: total tokens per second, or requests per second, that one server sustains across all users at once. Latency is a single user's experience; throughput is the machine's total output. They are not the same, and they are often in direct tension.
Cost is what each answer consumes: dollars per million tokens, memory in gigabytes to hold the weights and the KV cache, and energy per token. Cost is where quality and speed are ultimately paid for.
Why the axes trade off
The tie that binds all three is memory. During autoregressive decode, the model produces one token at a time, and every step must read the entire set of weights (and the growing KV cache) out of GPU memory into the compute cores. Pope et al. (2022), analysing PaLM inference on TPU v4, describe exactly this: the large memory footprint "gives rise to a large amount of memory traffic to load the parameters and KV cache from high-bandwidth memory (HBM) into the compute cores for each step," and they note that prefill "can run in parallel" over the input while "decode must run sequentially," so the two phases "have different performance characteristics." Decode is therefore memory-bandwidth-bound: the floor on inter-token latency is not how much arithmetic the GPU can do, but how fast it can stream the weights.
That single fact fuses quality, speed, and cost into one connected surface. Take a 70-billion-parameter model. In FP16 each weight occupies 16 bits = 2 bytes, so the weights alone are 70B × 2 bytes = 140 GB. If the GPU streams memory at roughly 3 TB/s, the fastest it can possibly emit one token is 140 GB ÷ 3,000 GB/s ≈ 47 ms — about 21 tokens per second for a single stream, before a single FLOP of useful math. To go faster on that hardware, you must move fewer bytes.
That is precisely what quantization does. Storing each weight in INT8 (8 bits = 1 byte) halves the footprint to 70 GB, which roughly halves the decode-latency floor to about 23 ms per token, doubles single-stream speed, and cuts the memory bill in half at the same time. Pope et al. reached a low-batch-size latency of 29 ms per token in their setup specifically by using int8 weight quantization. So one change buys speed and cost together. The catch lands on the quality axis: representing a weight in fewer bits rounds away information the model was using. Dettmers et al. (2022) showed this is not free at scale — naive INT8 quantization (absmax, row-wise, zeropoint) degrades once "systematic outliers occur at a scale of 6.7B parameters," pushing a 13B model's validation perplexity to 19.08 against a 12.45 full-precision baseline, until their outlier-aware LLM.int8() method restored near-lossless quality by isolating the rare outlier dimensions into 16-bit math. Quantization moves you along the frontier; it does not let you off it.
Latency versus throughput
Batching is the other lever, and it trades the two speed sub-axes against each other. Because a decode step already pays the full price of loading every weight, serving 32 users' tokens in one batched step reuses that same expensive weight-load 32 times over. So throughput — tokens per second across the fleet, and therefore cost per token — climbs almost linearly with batch size, until the GPU finally saturates its compute and becomes compute-bound rather than memory-bound. But a bigger batch makes each step take longer, and it forces early requests to wait while the batch fills, so any one user's latency gets worse. A chatbot tuned to feel instant runs small batches and eats a higher cost per token; a pipeline embedding a million documents overnight runs the largest batch that fits and eats the latency. Same model, same silicon, opposite settings — because they live at different corners of the frontier. Pushing the whole surface outward, rather than sliding along it, is the job of inference optimization.
Real-World Applications
The value of the frontier framing is that real teams resolve it differently for each workload, and the "right" performance target is set by who — or what — is waiting for the answer.
An interactive assistant streams tokens to a human who is reading them as they appear. The metrics that decide whether it feels good are time to first token and inter-token latency, not fleet throughput. Teams therefore stream output, run small batches, and accept a higher cost per token, because a reply that lands 200 ms sooner is worth more than one served more cheaply. The quality axis is capped only at "accurate enough to trust," beyond which extra accuracy that costs latency is a bad trade here.
An offline or batch pipeline — nightly embedding of a document corpus, bulk classification, dataset labeling, synthetic-data generation — has no human waiting. The target flips to throughput and cost per token, so these jobs run the biggest batch that fits and very often a quantized model, tolerating multi-second per-request latency that would be unacceptable in a chat window. The same accuracy that was "enough" for chat may be tightened here, because there is latency budget to spend on a larger or higher-precision model.
An on-device deployment — a phone, a car, a camera running Edge AI — has a hard, fixed budget on the cost axis: a set amount of memory and a battery. Here the binding constraint is that the model must fit and run at all without a network round-trip, so the decision is to quantize aggressively and pick a smaller model, explicitly trading quality away to buy the ability to run locally. Across all three, teams watch these numbers continuously in production with monitoring, because the frontier a system actually operates on drifts as load, prompt lengths, and hardware change.
Challenges
No single number is safe to optimize. Collapse "performance" to one metric and you will optimize the metric instead of the system. Chase a leaderboard score alone and you may ship a model too slow to serve; chase tokens per second alone and you may quantize past the accuracy your users actually needed. The frontier exists precisely so that a win reported on one axis is not a win overall — it has to be read together with what it cost the other two.
Scores are only comparable within one version of the test. A quality number carries an implicit "as measured by, and when." An Artificial Analysis Intelligence Index score, for example, is meaningless without its index version (such as v4.1), because the index is re-weighted over time and a figure from one version cannot be compared against another. Comparing two models on scores drawn from different index versions — or different benchmark revisions — is comparing nothing at all, even though both are real numbers.
The quantization cliff is not linear. INT8 is usually a favourable trade, but the Dettmers result is a warning that quality can fall off a cliff at scale rather than degrade gently, unless the rare outlier values are handled. Dropping to 4-bit and below widens the gap further and interacts with model architecture in ways that are hard to predict. "Just quantize it" is a decision on the quality axis, never a free lunch.
Latency and throughput cannot both be maximised on shared hardware. The very batching knob that lowers cost per token raises tail latency, and a serving system has to pick one operating point for both. That point is a product decision — how long a user will wait versus how much each answer may cost — not a purely technical one, and it usually has to hold under wildly varying load.
The frontier moves. Better kernels (FlashAttention-style memory-efficient attention), speculative decoding, and new hardware with more bandwidth push the whole surface outward, so a trade that was acceptable last year can be wasteful today. This is also why performance is monitored continuously rather than measured once: the position that was optimal is only optimal until the ground under it shifts.
Code Example
The decode-latency floor is worth computing yourself, because it explains the counter-intuitive result that a "faster" GPU with the same memory bandwidth generates text no faster. The floor is simply the bytes of weights that must be streamed per token, divided by memory bandwidth.
def decode_floor(n_params_billion, bytes_per_param, bandwidth_tb_s):
model_bytes = n_params_billion * 1e9 * bytes_per_param
bandwidth = bandwidth_tb_s * 1e12 # bytes per second
seconds_per_token = model_bytes / bandwidth
return seconds_per_token * 1e3, 1 / seconds_per_token
for label, bpp in [("FP16", 2), ("INT8", 1)]:
ms, tps = decode_floor(70, bpp, 3.0) # 70B model, ~3 TB/s
print(f"{label} ({bpp} B/param): {ms:.1f} ms/token, {tps:.1f} tokens/s")
Running it prints:
FP16 (2 B/param): 46.7 ms/token, 21.4 tokens/s
INT8 (1 B/param): 23.3 ms/token, 42.9 tokens/s
Halving the bytes per weight roughly halves the time per token and doubles single-stream speed — the quantization trade, made concrete. These are lower bounds: a real system also streams the KV cache, spends some time on compute, and adds framework overhead, so measured latency is always slower than the floor. But the floor is what sets the ceiling on speed, and it is fixed by memory bandwidth and model size — which is exactly why cutting the model's byte count is the highest-leverage move on the whole frontier.