--- source: 'https://howaiworks.ai/glossary/model-deployment' section: glossary title: Model Deployment description: >- What has to happen between a model that trains well and users calling it: replica sizing, latency budgets, cold starts, and the rollback nobody built. tags: - Inference - MLOps - DevOps - AI Infrastructure - LLM - GPU - Edge AI category: Machine Learning datePublished: '2025-07-16' lastUpdated: '2026-07-22' --- # Model Deployment > What has to happen between a model that trains well and users calling it: replica sizing, latency budgets, cold starts, and the rollback nobody built. ## Definition **Model deployment is everything between "the model scores well" and "other people are calling it" — and the arithmetic in that gap routinely says you need three to four times the hardware you estimated, and that 95% of your response time is a thing you cannot optimise away.** A trained model is a file. A deployed model is a service with a queue in front of it, a fixed amount of GPU memory behind it, a version you can put back when it misbehaves, and a bill that arrives whether or not anyone made a request. The gap is where most machine learning projects die, and not for mysterious reasons. The notebook ran one request at a time on a warm GPU, so it measured none of the four things that decide whether a deployment works: how long a request waits behind other requests, how much memory each concurrent user costs, what happens on the first request after the machine has been idle, and what you do at 02:00 when the new model is worse than the old one. This page does that arithmetic. The lifecycle *around* the deployment — versioning data, retraining cadence, drift — is [MLOps](https://howaiworks.ai/glossary/mlops); this page is the serving path itself. ## How It Works ### The five things that must exist before one request can be served Deploying a model is not one step, and naming the steps is the cheapest way to see which ones a notebook silently skipped. 1. **An artifact that contains more than weights.** The `.safetensors` file is not the model. The model is the weights *plus* the tokenizer or feature transform, plus the exact library versions that produce identical numerics. Ship the weights alone and someone reimplements the preprocessing in the request handler, which is the classic [training-serving skew](https://howaiworks.ai/glossary/mlops) bug. 2. **A server that holds the weights resident.** Loading 16 GB from disk per request is not a deployment. The server keeps the model in GPU memory, accepts many callers at once and batches them — the economics of that batch are the subject of [inference optimization](https://howaiworks.ai/glossary/inference-optimization). 3. **A route and an admission policy.** Something has to decide which replica gets the request, what happens when all of them are busy, and when to shed load instead of queueing it. 4. **A rollback path that exists before the first deploy.** Not "we could retrain" — a previous artifact, addressable by version, that a single command puts back. 5. **Observation that watches the model and not only the process.** A model server returns HTTP 200 while producing nonsense. See [monitoring](https://howaiworks.ai/glossary/monitoring). ### Where the response time actually goes Take a concrete request: a 2,000-token prompt, a 300-token answer, an 8-billion-parameter model in bfloat16 on one H100 SXM. The hardware numbers are NVIDIA's — **80 GB of HBM at 3.35 TB/s**, and 989 dense BF16 TFLOPS (the [datasheet](https://www.nvidia.com/en-us/data-center/h100/) lists 1,979 "with sparsity", so dense is half of that). Everything else below is an assumption, labelled as one. Prefill — reading the prompt — costs roughly 2 floating-point operations per parameter per token, so 2 x 8 x 10⁹ x 2,000 = **32 TFLOP**. Assume a realistic 40% of peak and it takes **81 ms**. Decode then emits one token at a time, and each step re-reads the weights and every batched user's cache: at a batch of 32 that is 33.2 GB per step, or **9.9 ms per token**. | Term | What it is | Time | |---|---|---| | Network round trip | client to region and back, connection already open | 40 ms | | Queue | waiting for a batch slot at moderate load | 30 ms | | Prefill | 2,000 prompt tokens at 2 FLOPs per parameter per token | 81 ms | | Decode | 300 output tokens at 9.9 ms each | 2,971 ms | | **Total** | | **3,122 ms** | **Decode is 95% of it.** That single ratio kills a large class of optimisation work before it starts: rewriting the handler in a faster language, moving the load balancer closer, shaving the JSON parsing — all of it competes for 5% of the budget. The two levers that matter are the batch size, which sets the 9.9 ms, and *how many tokens you let the model emit*, which is a product decision. Cutting the answer from 300 tokens to 150 halves the response time. No infrastructure change comes close. The same table also explains why streaming feels different. Time to first token is only 40 + 30 + 81 = **151 ms**. The user starts reading at 151 ms and the request finishes at 3.1 seconds, and the perceived latency is the first number. ### How many replicas: start from Little's Law, not throughput The instinct is to divide required tokens per second by tokens per second per GPU. Little's Law is the same calculation with a shape that is harder to get wrong: the number of requests in flight equals the arrival rate times the time each one stays. One GPU running a batch of 32 holds 32 requests in flight, and each holds its slot for 300 x 9.9 ms = **2.97 s**. So one GPU absorbs 32 / 2.97 = **10.8 requests per second**. A target of 40 requests per second needs **3.71 GPUs**. That is the number you must not provision. Three corrections, each of which is a real cost: - **Utilisation headroom.** Queueing delay is not linear in load. For a single-server queue the expected wait is `ρ/(1−ρ)` times the service time: **1x** at 50% utilisation, **4x** at 80%, **19x** at 95%. Real batched GPU servers are many-server queues and are kinder than that formula, but the shape is universal — the curve is a hyperbola and it goes vertical. Planning to 70% of capacity turns 3.71 into **5.31**. - **Rollout surge.** Kubernetes' default rolling update allows [25% max surge and 25% max unavailable](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/), meaning new pods start *before* old ones stop. On a GPU node pool with no spare capacity the new pods sit `Pending` and the rollout never completes. Budgeting the surge gives **6.63**. - **Zone redundancy.** If the service must survive losing one of two availability zones, each zone must be able to carry the whole load: **13.26**. Round up and the honest plan is **14 GPUs** where the first calculation said 4 — a **3.5x** gap, and the single most common reason a pilot that worked becomes a production system that cannot be afforded. The `## Code Example` at the end is this calculation, parameterised. ### The memory that is not in the model file Training memory is dominated by optimizer states and activations, which disappear at inference, so people expect serving to be the cheap side. Per copy of the model it is. But serving adds a term training does not have at all: a [KV cache](https://howaiworks.ai/glossary/kv-cache) per *concurrent user*, growing linearly with [context length](https://howaiworks.ai/glossary/context-window). For a Llama-3-8B-shaped model — [Table 3 of the Llama 3 paper](https://arxiv.org/abs/2407.21783) gives 32 layers, 8 key/value heads, and a model dimension of 4,096 across 32 attention heads, so a head dimension of 128 — the cache costs 2 x 32 x 8 x 128 x 2 bytes = **128 KiB per token**. So: - Weights in bfloat16: 8 x 10⁹ x 2 = **16 GB**. - 32 users at 4,096 tokens each: 32 x 4,096 x 128 KiB = **17.2 GB**. - Total resident: **33.2 GB** — comfortable on an 80 GB H100, and out of the question on a 24 GB card, where 16 GB of weights plus roughly 2 GB of CUDA context and activations leave 6 GB, or about **eleven** such sessions. Double the average context to 8,192 tokens and it is **five**. This is the concrete form of "it worked on my machine". The dev box ran one request at 2,000 tokens and used 16.3 GB. Production runs forty at 8,000 and needs 58 GB. Nothing about the model changed. ## Types The four deployment shapes below are distinguished by exactly two things — the latency the caller will tolerate, and what that tolerance costs per token — and every other difference follows. ### Batch inference Latency budget: hours to a day. Because nothing is waiting, the scheduler is free to run the largest batch memory allows, which puts the cost per token at its floor. Providers sell this directly: Anthropic's Message Batches API is [charged at 50% of the standard API prices](https://docs.claude.com/en/docs/build-with-claude/batch-processing) against a 24-hour expiry window, and OpenAI's Batch API works the same way. That discount is the throughput-latency tradeoff [priced as a product](https://howaiworks.ai/glossary/inference-optimization). Operationally it is a *job*, not a server — no autoscaler, no p99, and a failure is a retry rather than a page. ### Online request-response Latency budget: tens to hundreds of milliseconds, end to end. The budget caps the batch, and the capped batch is what you pay for: the same hardware costs several times more per token than in the batch regime. Deployment shape is a long-lived replica set behind a load balancer, sized by the arithmetic above. ### Streaming Same server as online, but the service level splits in two: time to first token and time per output token. Only the first is tight, which means the achievable batch is larger than a naive end-to-end SLO would allow — in the budget above, TTFT is 151 ms while the whole request is 3.1 seconds. Cost sits with online inference; the perceived latency sits with batch. ### Edge and on-device Batch size is 1 by construction, which is the worst point on the cost curve — but the marginal cost to *you* is zero, because the user's hardware pays it. The latency floor stops being the network round trip and becomes the device's memory bandwidth, which is typically an order of magnitude below a datacentre GPU, so time per token is far worse while time to first token can be better. This is the regime [Edge AI](https://howaiworks.ai/glossary/edge-ai) covers, and what local runtimes like [Ollama](https://howaiworks.ai/ai-tools/ollama) and [LM Studio](https://howaiworks.ai/ai-tools/lm-studio) implement. ## Real-World Applications **Google's TPU deployment is the clearest published measurement of what a latency target costs.** In [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://arxiv.org/abs/1704.04760) (ISCA 2017), the team reports Table 4 for a production MLP workload with a **99th-percentile response-time limit of 7 ms**, "required by the application developer". Unconstrained, the K80 GPU reached 36,465 inferences per second at batch 64 — but its p99 was 8.3 ms, over the limit. Forced back to batch 16 to meet 7 ms, it delivered **13,461 inferences per second: 37% of the throughput the same silicon could produce**. The CPU landed at 42%, the TPU at 80%. The deployment decision — which accelerator to buy — was not made on peak performance at all; it was made on performance *inside a latency budget*, and the two rankings differ. **vLLM is the reason a deployment choice, not a model choice, changes how many users a GPU holds.** The [PagedAttention paper](https://arxiv.org/abs/2309.06180) (SOSP 2023) profiled existing serving systems and found that only **20.4%–38.2% of KV cache memory actually held token states** — the rest was reservation and fragmentation. Managing that memory in pages instead of contiguous blocks took effective utilisation to **96.3%** and raised throughput **2–4x at the same latency**. Nobody retrained anything. Swapping the serving layer is often the largest single win available to a deployment, and it is invisible from inside the notebook. **Kubernetes is where most of these services actually run, and its defaults are deployment policy.** A `Deployment` with the default `RollingUpdate` strategy keeps [at least 75% of pods available and at most 125% running](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) during an update, and `kubectl rollout undo` reverts to the previous ReplicaSet. Those two defaults are the entire rollback story for a stateless web service — and they are *not* the rollback story for a model, because the ReplicaSet records the container image, not the weights the container downloads at start-up. **Knative and KServe make scale-to-zero a checkbox, which is why its arithmetic matters.** Knative's autoscaler defaults its [`scale-to-zero-pod-retention-period` to `0s`](https://knative.dev/docs/serving/autoscaling/scale-to-zero/) — once the autoscaler decides to scale to zero, the last pod is kept alive for no minimum time at all, bounded above by a `scale-to-zero-grace-period` of 30 seconds — and the next request after that pays a full cold start. For a CPU microservice that is a few hundred milliseconds. For a model server it is a container image plus 16 GB of weights pulled across the network: at an assumed 1 GB/s, **16 seconds**, or about **106x** the 151 ms time-to-first-token the same service promises when warm. ## Key Concepts - **Shadow deployment**: the new model receives a copy of real production traffic and its answers are discarded. It is the only test that uses the true input distribution while risking nothing, and its cost is one extra replica set for the duration. - **Canary**: a small slice of traffic where the answers *are* used, watched for a fixed window before the slice grows. The two parameters are slice size and window length, and they are set by the size of the regression you intend to catch. - **Blue-green**: two complete fleets, all traffic flipped at once. Rollback is a routing change measured in seconds, and the price is running double the GPUs until you delete the old fleet. - **Readiness versus liveness**: a replica still loading 16 GB of weights must fail its *readiness* probe, or the load balancer routes to it and those requests time out — and it must pass its *liveness* probe throughout, or the orchestrator kills it mid-load and restarts it forever. The two probes have opposite requirements during exactly the window a model server is slowest. - **Model version pinning**: the deployed artifact must be an immutable, addressable version, not a `latest` tag in a bucket. A mutable pointer means your rollback restores the pod spec and not the model. ## Challenges **The rollback that does not roll anything back.** A team ships version 2, sees it is worse, runs `kubectl rollout undo`, and gets the previous *container image* — which starts up and downloads `s3://models/production/latest`, the object that was overwritten an hour ago. The service is now running the bad model with the old image. This failure is specific to model deployment and it happens because the weights are the one input that is not in the repository, so ordinary deployment tooling never learns to version them. **A canary that cannot see the regression it was deployed to catch.** Suppose the baseline error rate is 1% and the canary takes 1% of 40 requests per second — 24 requests a minute. Over a ten minute window that is 240 requests: 2.4 expected errors with a standard deviation of 1.54. A model that fails **5%** of the time shows up at 6.2 standard deviations and is caught. A model that merely *doubles* the error rate to 2% shows 4.8 errors against 2.4 expected — **1.6 sigma**, which is noise. To see it at three sigma you need about **1,000 canary requests**, or 42 minutes. To see a rise from 1.0% to 1.2% you need **22,000 requests: over 15 hours**. Most teams pick 1% and ten minutes because both numbers sound cautious, and then ship regressions that were statistically invisible to the check that approved them. **Cold starts and cost savings are the same number.** With Poisson arrivals at rate λ and an idle timeout T, a request finds the instance cold exactly when the gap since the previous request exceeded T, with probability `e^(−λT)` — and the instance is billed for the remaining `1 − e^(−λT)` of the time. Those two sum to one. You cannot buy the saving without buying the misses in the same proportion: | Requests/day | λT (T = 5 min) | Requests hitting a cold start | Fraction of hours billed | |---|---|---|---| | 10,000 | 34.7 | ~0% | 100% | | 2,000 | 6.9 | 0.1% | 99.9% | | 500 | 1.7 | 17.6% | 82.4% | | 200 | 0.69 | 49.9% | 50.1% | | 50 | 0.17 | 84.1% | 15.9% | Read the 2,000/day row: scale-to-zero saves **one tenth of one percent**. Read the 200/day row: it halves the bill and half of your users wait 16 seconds. Lengthening T slides you up the table and shortening it slides you down; nothing moves you off the curve. Scale-to-zero is a development and preview-environment feature that gets deployed to production because it is on the same page of the console as the autoscaler. **The load test that measured your average request.** Synthetic benchmarks send fixed prompt and output lengths. Real traffic has output lengths spanning two orders of magnitude, and a single 8,000-token prompt admitted into the batch stalls every stream in it for the duration of its prefill. The mean time per token barely moves; the 99th percentile doubles. A deployment validated on mean latency will be signed off and then feel broken. **A healthy service and a wrong model look identical from outside.** Error rate zero, latency flat, CPU normal — and the input distribution moved last Tuesday. Deployment tooling has no probe for this, because there is nothing to probe: the failure is a change in a distribution, not in a response code. That is why the deployment checklist has to include an [observability](https://howaiworks.ai/glossary/monitoring) plan for the model's own outputs, and why the [MLOps](https://howaiworks.ai/glossary/mlops) lifecycle exists around the serving path rather than inside it. ## Future Trends The cold-start term is the one under active attack, and if it falls the arithmetic above changes shape. Snapshot-and-restore of GPU memory — checkpointing a warmed process and mapping it back rather than re-reading weights from storage — turns a 16-second start into something closer to a container start. The moment the cold start is smaller than the request it interrupts, the `e^(−λT)` trade stops being a trade, and scale-to-zero becomes the default for everything below continuous traffic rather than a trap. The unit of deployment is also drifting away from "a replica running a model". Prefill and decode are already being run on separate hardware pools by large serving stacks, so a replica is two things with a cache shipped between them. Above that, routers that pick a model per request make the deployed artifact a *policy* rather than a binary, and the rollback unit becomes a routing rule — faster and cheaper to revert than any container, and much easier to change by accident. Catalog services like [OpenRouter](https://howaiworks.ai/ai-tools/openrouter) are early versions of this, and the operational question they raise is new: what does "which model answered my request last Tuesday?" mean, and can you still answer it. The third shift is that the cheapest deployment target keeps moving toward the caller. Quantised models small enough to run in a browser or on a phone flip the cost model from per-token infrastructure you own to zero marginal cost on hardware you do not — while flipping the service level from something you control to whatever the user's device manages. That is not a better deployment; it is a different one, with the memory-bandwidth arithmetic of this page applied to a device with roughly a tenth of the bandwidth. ## Code Example The capacity plan above, parameterised. Change the assumptions at the top and it re-derives the fleet — the point is that the last number is several times the first, always, for structural reasons: ```python import math # ---- assumptions. Change these; the arithmetic is the point. ---------------- TARGET_QPS = 40 # peak requests per second the service must absorb OUT_TOKENS = 300 # tokens in an average answer BATCH = 32 # concurrent sequences per GPU, chosen for latency CTX = 4096 # average tokens held in one sequence's cache UTIL = 0.70 # plan to this load, not to 100% -- queues explode near 1 SURGE = 0.25 # Kubernetes RollingUpdate default maxSurge ZONES = 2 # and the service must survive losing one of them PARAMS, WBYTES = 8e9, 2 # 8B parameters, bfloat16 LAYERS, KVH, HD = 32, 8, 128 # Llama-3-8B shape HBM, BW = 80e9, 3.35e12 # H100 SXM: 80 GB at 3.35 TB/s # ---- one GPU ----------------------------------------------------------------- kv_tok = 2 * LAYERS * KVH * HD * WBYTES # bytes of KV cache per token weights = PARAMS * WBYTES cache = BATCH * CTX * kv_tok step = weights + cache # a decode step reads both tpot = step / BW # seconds per token, for the batch service = OUT_TOKENS * tpot # how long a request holds a slot per_gpu = BATCH / service # Little's Law: lambda = L / W print(f"KV cache {kv_tok/1024:>8.0f} KiB per token") print(f"decode step {step/1e9:>8.1f} GB read ({weights/1e9:.0f} weights" f" + {cache/1e9:.1f} cache)") print(f"memory used {step/1e9:>8.1f} GB of {HBM/1e9:.0f} GB") print(f"TPOT {tpot*1e3:>8.1f} ms per token") print(f"one request {service:>8.2f} s of GPU slot") print(f"one GPU serves {per_gpu:>8.2f} requests/second\n") # ---- the fleet --------------------------------------------------------------- naive = TARGET_QPS / per_gpu headroom = naive / UTIL rollout = headroom * (1 + SURGE) resilient = rollout * ZONES / (ZONES - 1) for label, n in [("throughput only", naive), (f"at {UTIL:.0%} utilisation", headroom), (f"+ {SURGE:.0%} rollout surge", rollout), (f"+ survive 1 of {ZONES} zones", resilient)]: print(f"{label:<26} {n:>6.2f} GPUs -> provision {math.ceil(n)}") print(f"\nplan / notebook estimate: {math.ceil(resilient)/math.ceil(naive):.2f}x") ``` Output: ``` KV cache 128 KiB per token decode step 33.2 GB read (16 weights + 17.2 cache) memory used 33.2 GB of 80 GB TPOT 9.9 ms per token one request 2.97 s of GPU slot one GPU serves 10.77 requests/second throughput only 3.71 GPUs -> provision 4 at 70% utilisation 5.31 GPUs -> provision 6 + 25% rollout surge 6.63 GPUs -> provision 7 + survive 1 of 2 zones 13.26 GPUs -> provision 14 plan / notebook estimate: 3.50x ``` The memory line is worth staring at: 33.2 GB of an 80 GB card. There is room for a much larger batch, and taking it would raise throughput and cut the fleet — at the price of a longer time per token for every user in it. That dial, and where to stop turning it, is [inference optimization](https://howaiworks.ai/glossary/inference-optimization). This page's job is the number underneath it: whatever batch you choose, the fleet you must actually buy is not the one the throughput division gave you. ## Frequently Asked Questions ### What is model deployment in simple terms? It is everything between a model that scores well in a notebook and a model that answers other people's requests: packaging the weights and the preprocessing into one artifact, running it behind a service that many callers hit at once, sizing enough hardware to absorb peak traffic, and keeping a way to put the previous version back. The training is the part that is finished; the deployment is the part that runs forever. ### Why is my model slower in production than it was in my notebook? Because your notebook ran one request at a time with the weights already resident in GPU memory. Production adds a network round trip, a queue behind other users, and — for a generative model — hundreds of sequential decode steps that each re-read the whole model. In the worked budget on this page, an 8B model answering a 2,000-token prompt with 300 tokens spends 81 ms on the prompt and 2,971 ms emitting the answer. Nothing you do to your web framework touches the 95% of the time that is decode. ### How many GPUs do I need to serve a model? Start from Little's Law rather than from throughput. If a request holds a batch slot for 2.97 seconds and one GPU runs 32 slots, that GPU absorbs 10.8 requests per second. Forty requests per second therefore needs 3.7 GPUs of raw capacity — but 6 to run at 70% utilisation, 7 to have somewhere to put new pods during a rolling update, and 14 to survive losing an availability zone. The last number is the one you buy. ### Does a model that fits on one GPU in training fit on one GPU in serving? Not necessarily, because serving adds a term training does not have: a key-value cache per concurrent user, which grows with context length. An 8B model in bfloat16 is 16 GB of weights, but its cache costs 128 KiB per token — so 32 users at 4,096 tokens each add another 17 GB. On a 24 GB card the weights fit and the service does not. ### What is the difference between shadow, canary and blue-green deployment? Shadow sends real traffic to the new model and throws its answers away, so you learn how it behaves without any user seeing it. Canary sends a small slice of real traffic and does act on the answers. Blue-green runs two complete fleets and flips all traffic at once, which makes rollback instant and doubles your hardware bill for the duration. ### Is serverless, scale-to-zero GPU inference worth it? Only at traffic so low that you will feel the cold starts. With Poisson arrivals and an idle timeout T, the fraction of requests that arrive to a cold instance and the fraction of hourly cost you avoid are the same number, e raised to minus lambda-T. At 200 requests a day with a five-minute timeout you save about half your GPU bill and half your users wait through a cold start. There is no setting where it is both cheap and fast. ## Related ### Related terms - [Machine Learning Operations (MLOps)](https://howaiworks.ai/glossary/mlops) - [Inference Optimization](https://howaiworks.ai/glossary/inference-optimization) - [KV Cache](https://howaiworks.ai/glossary/kv-cache) - [Production Systems](https://howaiworks.ai/glossary/production-systems) - [Monitoring](https://howaiworks.ai/glossary/monitoring) - [Inference](https://howaiworks.ai/glossary/inference) --- Source: https://howaiworks.ai/glossary/model-deployment — HowAIWorks.ai