Model Deployment

Definition

Model deployment is everything between "the model scores well" and "other people are calling it" — and the arithmetic in that gap routinely says you need three to four times the hardware you estimated, and that 95% of your response time is a thing you cannot optimise away. A trained model is a file. A deployed model is a service with a queue in front of it, a fixed amount of GPU memory behind it, a version you can put back when it misbehaves, and a bill that arrives whether or not anyone made a request.

The gap is where most machine learning projects die, and not for mysterious reasons. The notebook ran one request at a time on a warm GPU, so it measured none of the four things that decide whether a deployment works: how long a request waits behind other requests, how much memory each concurrent user costs, what happens on the first request after the machine has been idle, and what you do at 02:00 when the new model is worse than the old one. This page does that arithmetic. The lifecycle around the deployment — versioning data, retraining cadence, drift — is MLOps; this page is the serving path itself.

How It Works

The five things that must exist before one request can be served

Deploying a model is not one step, and naming the steps is the cheapest way to see which ones a notebook silently skipped.

1. An artifact that contains more than weights. The .safetensors file is not the model. The model is the weights plus the tokenizer or feature transform, plus the exact library versions that produce identical numerics. Ship the weights alone and someone reimplements the preprocessing in the request handler, which is the classic training-serving skew bug.
2. A server that holds the weights resident. Loading 16 GB from disk per request is not a deployment. The server keeps the model in GPU memory, accepts many callers at once and batches them — the economics of that batch are the subject of inference optimization.
3. A route and an admission policy. Something has to decide which replica gets the request, what happens when all of them are busy, and when to shed load instead of queueing it.
4. A rollback path that exists before the first deploy. Not "we could retrain" — a previous artifact, addressable by version, that a single command puts back.
5. Observation that watches the model and not only the process. A model server returns HTTP 200 while producing nonsense. See monitoring.

Where the response time actually goes

Take a concrete request: a 2,000-token prompt, a 300-token answer, an 8-billion-parameter model in bfloat16 on one H100 SXM. The hardware numbers are NVIDIA's — 80 GB of HBM at 3.35 TB/s, and 989 dense BF16 TFLOPS (the datasheet lists 1,979 "with sparsity", so dense is half of that). Everything else below is an assumption, labelled as one.

Prefill — reading the prompt — costs roughly 2 floating-point operations per parameter per token, so 2 x 8 x 10⁹ x 2,000 = 32 TFLOP. Assume a realistic 40% of peak and it takes 81 ms. Decode then emits one token at a time, and each step re-reads the weights and every batched user's cache: at a batch of 32 that is 33.2 GB per step, or 9.9 ms per token.

Decode is 95% of it. That single ratio kills a large class of optimisation work before it starts: rewriting the handler in a faster language, moving the load balancer closer, shaving the JSON parsing — all of it competes for 5% of the budget. The two levers that matter are the batch size, which sets the 9.9 ms, and how many tokens you let the model emit, which is a product decision. Cutting the answer from 300 tokens to 150 halves the response time. No infrastructure change comes close.

The same table also explains why streaming feels different. Time to first token is only 40 + 30 + 81 = 151 ms. The user starts reading at 151 ms and the request finishes at 3.1 seconds, and the perceived latency is the first number.

How many replicas: start from Little's Law, not throughput

The instinct is to divide required tokens per second by tokens per second per GPU. Little's Law is the same calculation with a shape that is harder to get wrong: the number of requests in flight equals the arrival rate times the time each one stays.

One GPU running a batch of 32 holds 32 requests in flight, and each holds its slot for 300 x 9.9 ms = 2.97 s. So one GPU absorbs 32 / 2.97 = 10.8 requests per second. A target of 40 requests per second needs 3.71 GPUs.

That is the number you must not provision. Three corrections, each of which is a real cost:

• Utilisation headroom. Queueing delay is not linear in load. For a single-server queue the expected wait is ρ/(1−ρ) times the service time: 1x at 50% utilisation, 4x at 80%, 19x at 95%. Real batched GPU servers are many-server queues and are kinder than that formula, but the shape is universal — the curve is a hyperbola and it goes vertical. Planning to 70% of capacity turns 3.71 into 5.31.
• Rollout surge. Kubernetes' default rolling update allows 25% max surge and 25% max unavailable, meaning new pods start before old ones stop. On a GPU node pool with no spare capacity the new pods sit Pending and the rollout never completes. Budgeting the surge gives 6.63.
• Zone redundancy. If the service must survive losing one of two availability zones, each zone must be able to carry the whole load: 13.26.

Round up and the honest plan is 14 GPUs where the first calculation said 4 — a 3.5x gap, and the single most common reason a pilot that worked becomes a production system that cannot be afforded. The ## Code Example at the end is this calculation, parameterised.

The memory that is not in the model file

Training memory is dominated by optimizer states and activations, which disappear at inference, so people expect serving to be the cheap side. Per copy of the model it is. But serving adds a term training does not have at all: a KV cache per concurrent user, growing linearly with context length.

For a Llama-3-8B-shaped model — Table 3 of the Llama 3 paper gives 32 layers, 8 key/value heads, and a model dimension of 4,096 across 32 attention heads, so a head dimension of 128 — the cache costs 2 x 32 x 8 x 128 x 2 bytes = 128 KiB per token. So:

• Weights in bfloat16: 8 x 10⁹ x 2 = 16 GB.
• 32 users at 4,096 tokens each: 32 x 4,096 x 128 KiB = 17.2 GB.
• Total resident: 33.2 GB — comfortable on an 80 GB H100, and out of the question on a 24 GB card, where 16 GB of weights plus roughly 2 GB of CUDA context and activations leave 6 GB, or about eleven such sessions. Double the average context to 8,192 tokens and it is five.

This is the concrete form of "it worked on my machine". The dev box ran one request at 2,000 tokens and used 16.3 GB. Production runs forty at 8,000 and needs 58 GB. Nothing about the model changed.

Types

The four deployment shapes below are distinguished by exactly two things — the latency the caller will tolerate, and what that tolerance costs per token — and every other difference follows.

Batch inference

Latency budget: hours to a day. Because nothing is waiting, the scheduler is free to run the largest batch memory allows, which puts the cost per token at its floor. Providers sell this directly: Anthropic's Message Batches API is charged at 50% of the standard API prices against a 24-hour expiry window, and OpenAI's Batch API works the same way. That discount is the throughput-latency tradeoff priced as a product. Operationally it is a job, not a server — no autoscaler, no p99, and a failure is a retry rather than a page.

Online request-response

Latency budget: tens to hundreds of milliseconds, end to end. The budget caps the batch, and the capped batch is what you pay for: the same hardware costs several times more per token than in the batch regime. Deployment shape is a long-lived replica set behind a load balancer, sized by the arithmetic above.

Streaming

Same server as online, but the service level splits in two: time to first token and time per output token. Only the first is tight, which means the achievable batch is larger than a naive end-to-end SLO would allow — in the budget above, TTFT is 151 ms while the whole request is 3.1 seconds. Cost sits with online inference; the perceived latency sits with batch.

Edge and on-device

Batch size is 1 by construction, which is the worst point on the cost curve — but the marginal cost to you is zero, because the user's hardware pays it. The latency floor stops being the network round trip and becomes the device's memory bandwidth, which is typically an order of magnitude below a datacentre GPU, so time per token is far worse while time to first token can be better. This is the regime Edge AI covers, and what local runtimes like Ollama and LM Studio implement.

Real-World Applications

Google's TPU deployment is the clearest published measurement of what a latency target costs. In In-Datacenter Performance Analysis of a Tensor Processing Unit (ISCA 2017), the team reports Table 4 for a production MLP workload with a 99th-percentile response-time limit of 7 ms, "required by the application developer". Unconstrained, the K80 GPU reached 36,465 inferences per second at batch 64 — but its p99 was 8.3 ms, over the limit. Forced back to batch 16 to meet 7 ms, it delivered 13,461 inferences per second: 37% of the throughput the same silicon could produce. The CPU landed at 42%, the TPU at 80%. The deployment decision — which accelerator to buy — was not made on peak performance at all; it was made on performance inside a latency budget, and the two rankings differ.

vLLM is the reason a deployment choice, not a model choice, changes how many users a GPU holds. The PagedAttention paper (SOSP 2023) profiled existing serving systems and found that only 20.4%–38.2% of KV cache memory actually held token states — the rest was reservation and fragmentation. Managing that memory in pages instead of contiguous blocks took effective utilisation to 96.3% and raised throughput 2–4x at the same latency. Nobody retrained anything. Swapping the serving layer is often the largest single win available to a deployment, and it is invisible from inside the notebook.

Kubernetes is where most of these services actually run, and its defaults are deployment policy. A Deployment with the default RollingUpdate strategy keeps at least 75% of pods available and at most 125% running during an update, and kubectl rollout undo reverts to the previous ReplicaSet. Those two defaults are the entire rollback story for a stateless web service — and they are not the rollback story for a model, because the ReplicaSet records the container image, not the weights the container downloads at start-up.

Knative and KServe make scale-to-zero a checkbox, which is why its arithmetic matters. Knative's autoscaler defaults its scale-to-zero-pod-retention-period to 0s — once the autoscaler decides to scale to zero, the last pod is kept alive for no minimum time at all, bounded above by a scale-to-zero-grace-period of 30 seconds — and the next request after that pays a full cold start. For a CPU microservice that is a few hundred milliseconds. For a model server it is a container image plus 16 GB of weights pulled across the network: at an assumed 1 GB/s, 16 seconds, or about 106x the 151 ms time-to-first-token the same service promises when warm.

Key Concepts

• Shadow deployment: the new model receives a copy of real production traffic and its answers are discarded. It is the only test that uses the true input distribution while risking nothing, and its cost is one extra replica set for the duration.
• Canary: a small slice of traffic where the answers are used, watched for a fixed window before the slice grows. The two parameters are slice size and window length, and they are set by the size of the regression you intend to catch.
• Blue-green: two complete fleets, all traffic flipped at once. Rollback is a routing change measured in seconds, and the price is running double the GPUs until you delete the old fleet.
• Readiness versus liveness: a replica still loading 16 GB of weights must fail its readiness probe, or the load balancer routes to it and those requests time out — and it must pass its liveness probe throughout, or the orchestrator kills it mid-load and restarts it forever. The two probes have opposite requirements during exactly the window a model server is slowest.
• Model version pinning: the deployed artifact must be an immutable, addressable version, not a latest tag in a bucket. A mutable pointer means your rollback restores the pod spec and not the model.

Challenges

The rollback that does not roll anything back. A team ships version 2, sees it is worse, runs kubectl rollout undo, and gets the previous container image — which starts up and downloads s3://models/production/latest, the object that was overwritten an hour ago. The service is now running the bad model with the old image. This failure is specific to model deployment and it happens because the weights are the one input that is not in the repository, so ordinary deployment tooling never learns to version them.

A canary that cannot see the regression it was deployed to catch. Suppose the baseline error rate is 1% and the canary takes 1% of 40 requests per second — 24 requests a minute. Over a ten minute window that is 240 requests: 2.4 expected errors with a standard deviation of 1.54. A model that fails 5% of the time shows up at 6.2 standard deviations and is caught. A model that merely doubles the error rate to 2% shows 4.8 errors against 2.4 expected — 1.6 sigma, which is noise. To see it at three sigma you need about 1,000 canary requests, or 42 minutes. To see a rise from 1.0% to 1.2% you need 22,000 requests: over 15 hours. Most teams pick 1% and ten minutes because both numbers sound cautious, and then ship regressions that were statistically invisible to the check that approved them.

Cold starts and cost savings are the same number. With Poisson arrivals at rate λ and an idle timeout T, a request finds the instance cold exactly when the gap since the previous request exceeded T, with probability e^(−λT) — and the instance is billed for the remaining 1 − e^(−λT) of the time. Those two sum to one. You cannot buy the saving without buying the misses in the same proportion:

Read the 2,000/day row: scale-to-zero saves one tenth of one percent. Read the 200/day row: it halves the bill and half of your users wait 16 seconds. Lengthening T slides you up the table and shortening it slides you down; nothing moves you off the curve. Scale-to-zero is a development and preview-environment feature that gets deployed to production because it is on the same page of the console as the autoscaler.

The load test that measured your average request. Synthetic benchmarks send fixed prompt and output lengths. Real traffic has output lengths spanning two orders of magnitude, and a single 8,000-token prompt admitted into the batch stalls every stream in it for the duration of its prefill. The mean time per token barely moves; the 99th percentile doubles. A deployment validated on mean latency will be signed off and then feel broken.

A healthy service and a wrong model look identical from outside. Error rate zero, latency flat, CPU normal — and the input distribution moved last Tuesday. Deployment tooling has no probe for this, because there is nothing to probe: the failure is a change in a distribution, not in a response code. That is why the deployment checklist has to include an observability plan for the model's own outputs, and why the MLOps lifecycle exists around the serving path rather than inside it.

Future Trends

The cold-start term is the one under active attack, and if it falls the arithmetic above changes shape. Snapshot-and-restore of GPU memory — checkpointing a warmed process and mapping it back rather than re-reading weights from storage — turns a 16-second start into something closer to a container start. The moment the cold start is smaller than the request it interrupts, the e^(−λT) trade stops being a trade, and scale-to-zero becomes the default for everything below continuous traffic rather than a trap.

The unit of deployment is also drifting away from "a replica running a model". Prefill and decode are already being run on separate hardware pools by large serving stacks, so a replica is two things with a cache shipped between them. Above that, routers that pick a model per request make the deployed artifact a policy rather than a binary, and the rollback unit becomes a routing rule — faster and cheaper to revert than any container, and much easier to change by accident. Catalog services like OpenRouter are early versions of this, and the operational question they raise is new: what does "which model answered my request last Tuesday?" mean, and can you still answer it.

The third shift is that the cheapest deployment target keeps moving toward the caller. Quantised models small enough to run in a browser or on a phone flip the cost model from per-token infrastructure you own to zero marginal cost on hardware you do not — while flipping the service level from something you control to whatever the user's device manages. That is not a better deployment; it is a different one, with the memory-bandwidth arithmetic of this page applied to a device with roughly a tenth of the bandwidth.

Code Example

The capacity plan above, parameterised. Change the assumptions at the top and it re-derives the fleet — the point is that the last number is several times the first, always, for structural reasons:

import math

# ---- assumptions. Change these; the arithmetic is the point. ----------------
TARGET_QPS = 40        # peak requests per second the service must absorb
OUT_TOKENS = 300       # tokens in an average answer
BATCH      = 32        # concurrent sequences per GPU, chosen for latency
CTX        = 4096      # average tokens held in one sequence's cache
UTIL       = 0.70      # plan to this load, not to 100% -- queues explode near 1
SURGE      = 0.25      # Kubernetes RollingUpdate default maxSurge
ZONES      = 2         # and the service must survive losing one of them

PARAMS, WBYTES   = 8e9, 2          # 8B parameters, bfloat16
LAYERS, KVH, HD  = 32, 8, 128      # Llama-3-8B shape
HBM, BW          = 80e9, 3.35e12   # H100 SXM: 80 GB at 3.35 TB/s

# ---- one GPU -----------------------------------------------------------------
kv_tok  = 2 * LAYERS * KVH * HD * WBYTES      # bytes of KV cache per token
weights = PARAMS * WBYTES
cache   = BATCH * CTX * kv_tok
step    = weights + cache                     # a decode step reads both
tpot    = step / BW                           # seconds per token, for the batch
service = OUT_TOKENS * tpot                   # how long a request holds a slot
per_gpu = BATCH / service                     # Little's Law: lambda = L / W

print(f"KV cache        {kv_tok/1024:>8.0f} KiB per token")
print(f"decode step     {step/1e9:>8.1f} GB read  ({weights/1e9:.0f} weights"
      f" + {cache/1e9:.1f} cache)")
print(f"memory used     {step/1e9:>8.1f} GB of {HBM/1e9:.0f} GB")
print(f"TPOT            {tpot*1e3:>8.1f} ms per token")
print(f"one request     {service:>8.2f} s of GPU slot")
print(f"one GPU serves  {per_gpu:>8.2f} requests/second\n")

# ---- the fleet ---------------------------------------------------------------
naive     = TARGET_QPS / per_gpu
headroom  = naive / UTIL
rollout   = headroom * (1 + SURGE)
resilient = rollout * ZONES / (ZONES - 1)

for label, n in [("throughput only", naive),
                 (f"at {UTIL:.0%} utilisation", headroom),
                 (f"+ {SURGE:.0%} rollout surge", rollout),
                 (f"+ survive 1 of {ZONES} zones", resilient)]:
    print(f"{label:<26} {n:>6.2f} GPUs  ->  provision {math.ceil(n)}")

print(f"\nplan / notebook estimate: {math.ceil(resilient)/math.ceil(naive):.2f}x")

Output:

KV cache             128 KiB per token
decode step         33.2 GB read  (16 weights + 17.2 cache)
memory used         33.2 GB of 80 GB
TPOT                 9.9 ms per token
one request         2.97 s of GPU slot
one GPU serves     10.77 requests/second

throughput only              3.71 GPUs  ->  provision 4
at 70% utilisation           5.31 GPUs  ->  provision 6
+ 25% rollout surge          6.63 GPUs  ->  provision 7
+ survive 1 of 2 zones      13.26 GPUs  ->  provision 14

plan / notebook estimate: 3.50x

The memory line is worth staring at: 33.2 GB of an 80 GB card. There is room for a much larger batch, and taking it would raise throughput and cut the fleet — at the price of a longer time per token for every user in it. That dial, and where to stop turning it, is inference optimization. This page's job is the number underneath it: whatever batch you choose, the fleet you must actually buy is not the one the throughput division gave you.