Definition
AI architecture, in the sense engineers mean today, is the system design of a production AI application: how retrieval, context assembly, the model call, tool execution, output validation, logging and evaluation are arranged around a model to make it a product. The model is the smallest part — one HTTPS request in a path where most of the code, most of the latency the user feels and nearly all of the failure modes live outside that request.
The same phrase also names the internal design of a network — layers, attention, expert routing. For that sense, see neural network, transformer, attention mechanism and mixture of experts. This page covers the other meaning: what you build around a model you did not train. Products rarely fail on the wrong number of attention heads; they fail because history overflowed the context window, or a tool returned an error the model narrated confidently.
How It Works
A request through a mature AI system touches the model briefly and everything else at length. The message arrives; the system embeds it, runs a vector search over your documents and reranks the hits; it assembles a prompt from system instructions, tool definitions, passages and as much history as fits; it calls the model; if the model asks for a tool, the system runs it (function calling), appends the result and calls again; it validates the output against a schema and logs the trace for tomorrow's evaluation job. Seven components, one of which is the model.
The organising constraint is statelessness. The model remembers nothing between calls, so conversation state lives in your system and every call re-sends everything the model must know. Context assembly becomes a component with a budget, and the budget is easy to blow. Take a 128,000-token window, a 2,000-token system prompt with tool definitions, and 6,000 tokens of retrieved documents. Suppose the agent queries a database each turn and appends the raw result — 200 rows of JSON, roughly 6,000 tokens. After twenty turns the transcript holds 20 × 6,000 = 120,000 tokens of tool output alone; with the system prompt and passages that is 128,000 exactly. The window is full on turn twenty.
Eviction policy is therefore an architectural decision. Summarising each tool result at the boundary — 200 rows down to a 300-token digest, the full result kept in your own store and re-fetchable by ID — costs one cheap model call per turn and drops those twenty turns from 120,000 tokens to 6,000, a twentyfold reduction. Whichever policy you pick, you choose what the model forgets first; if you have not chosen, your framework has.
The other decision you live with forever is retrieval versus fine-tuning. Knowledge that changes, or must be cited, belongs in retrieval: a document updated this morning is live this afternoon and can be shown as a source. Behaviour and format — this JSON shape, this refusal tone, this vocabulary — belong in the weights, being stable and expensive to restate on every call. Teams that fine-tune facts retrain to fix a typo; teams that prompt their way to a rigid format pay for a 3,000-token system prompt forever.
Real-World Applications
Code completion is the clearest case of the model being the small part. A Copilot-style assistant spends most of its engineering effort deciding what to send: the text either side of the cursor plus snippets from recently opened files, ranked and trimmed to a prompt budget within the few hundred milliseconds a developer tolerates between keystrokes. That retrieval must run in the editor, because the source of truth is an unsaved buffer — where the data lives dictates the topology.
Citation-grounded search assistants turn that rule into a hard requirement. A product showing a link beside every claim cannot get it from weights, which carry no provenance; the document must be in context at generation time. That forces a retrieval layer, an index refresh pipeline, and a check that each cited URL really appeared in the retrieved set — telling a model to "only use the sources provided" without verifying afterwards is a wish, not a control.
The Model Context Protocol, an open standard released in November 2024, moved a third piece: a tool integration used to be code inside your assistant and is now a separate process any client connects to, which is why agentic products ship server catalogues rather than plugin folders.
Key Concepts
Latency budgets compose by addition. A chained request waits for the sum of its parts, rarely where teams look. Take 20 ms to embed the query, 30 ms for the vector search, 80 ms to rerank, a first model call at 400 ms to first token plus 300 output tokens at 60 tokens per second (5.0 s), a 250 ms internal API call, then a second model call at 400 ms plus 200 tokens (3.7 s): about 9.5 seconds. The retrieval stack the team spent a month tuning is 130 ms of that, or 1.4%; the two generations are 9.1 s, or 96%. Halving retrieval saves 65 ms; dropping the second model call saves 3.7 seconds.
Cost is an input-token problem. Assume $3 per million input tokens and $15 per million output. That request sends 28,000 input tokens on the first call and 34,000 on the second (everything again, plus the tool result): 62,000 × $3/1M = $0.186, plus 500 output tokens × $15/1M = $0.008 — about $0.19 a request, 96% of it input. At 5,000 requests a day, $970 a day and $29,000 a month. Cutting retrieval from twelve chunks to six removes 6,000 tokens across the two calls and 9% of the bill; halving the answer removes under 2%.
Evaluation is a component, not a phase. Output is non-deterministic, so the usual regression test — same input, same expected output — does not exist here. A stored set of inputs with graded outputs replaces it, run on every prompt or model change. Size it honestly: on a 200-example set, 82% to 85% is six more examples passing, with a standard error around ±3.6 percentage points — the gain is inside the noise. Resolving three points needs roughly a thousand examples, or a paired run scoring both versions on the same items.
Validate output rather than trying to prevent bad generations. Checking that a response parses as the schema you asked for, that cited IDs exist and that no field violates policy is cheap, deterministic and testable; prompting a model into never producing bad output is none of those. A failed check costs one retry; a hallucinated field reaching the user costs trust. Build that retry loop deliberately: retrying everything doubles latency and cost for the slowest requests.
Challenges
Non-determinism removes the safety net: no golden-file test, no reproducible bug report, no clean bisect, because one input can produce a good answer and a bad one an hour apart. Debugging becomes trace archaeology, which is why logging the full request — prompt, chunks, tool calls, raw output — is load-bearing infrastructure, not an observability nicety.
Failures are also partial and quiet. A truncated context does not throw; it makes the answer worse. A tool returning an empty list does not error; the model narrates the emptiness as fact. Schema validation catches shape, never truth, so a valid JSON object can carry a fabricated order number.
Model upgrades are breaking changes with no compiler to catch them: a new snapshot can shift formatting, refusal boundaries and tool-calling style at once, nothing in the build fails, and the eval set is the only thing that will tell you. Prompt caching hides the same coupling — providers cache on a stable prefix, so putting retrieved documents ahead of the system prompt destroys the hit rate.
Components compound, too: four services at 99.5% availability each are 98% end to end, one failed request in fifty before the model has said anything wrong — which is why timeouts, a fallback that answers without retrieval, and a circuit breaker that fails a feature rather than a page earn their keep.
Future Trends
Longer context windows will not delete the retrieval layer; they relocate its cost. Filling a million-token window is possible and economically silly — at $3 per million input tokens that is $3 per request before a token is generated, plus seconds of prefill. Retrieval survives as cost control, and the work moves to caching prompt prefixes that do not change between calls.
Model routing is becoming a first-class component. When a small model answers 70% of requests at a tenth of the cost and escalates the rest, the blended cost of that $0.19 request falls to about $0.07 — and the router, a classifier with its own accuracy, latency and eval set, is yours to own. Evaluation moves into CI beside it as a gate: a prompt change that drops the score blocks the merge, which is roughly where this starts to feel like software engineering again.