API

Definition

An API (Application Programming Interface) is the published contract by which one program calls another. In AI that contract is almost always an HTTPS endpoint you send a JSON body to: you post some text, a machine elsewhere performs a few hundred billion multiplications, and text comes back. Almost nobody runs a frontier large language model themselves, so the API is the product.

What earns a model API its own entry is that it breaks three assumptions every developer brings from ordinary web services. It is billed by the token rather than by the request, so two calls to the same URL can differ in cost a thousandfold. It is stateless: the model remembers nothing between calls, so a "conversation" is your own client resending the entire transcript every turn. And the same request sent twice can legitimately return different text, which quietly invalidates response caching, safe retries and most test assertions. Miss any one of those and you ship a bug or a surprise invoice.

How It Works

A request carries a model identifier, a list of messages with roles, and sampling settings such as temperature. The server turns the messages into token ids and runs the model, and that work splits into two phases with very different physics. Prefill processes the whole prompt in one parallel pass, because all of its tokens are known at once. Decode then produces the reply one token at a time, each token needing its own full pass through the model, because token 400 cannot be computed before token 399 exists. Decode is limited by how fast weights can be read from memory rather than by arithmetic — a 70-billion parameter model in 8-bit precision means reading roughly 70 GB per token, which on a 3.35 TB/s accelerator caps a single stream near 47 tokens per second (see the memory wall). That asymmetry, not vendor margin, is why output tokens cost several times what input tokens cost.

Statelessness is the property that costs people real money, so do the arithmetic. Suppose each turn of a chat adds 500 tokens — a 100-token question and a 400-token answer. On turn n your client sends everything said so far plus the new question: 500 × (n − 1) + 100 tokens. Across 20 turns that totals 500 × 190 + 2,000 = 97,000 input tokens billed to hold a conversation whose full text is only 10,000 tokens, so you paid for the transcript nearly ten times over. Because the sum is n(n−1)/2, doubling the chat to 40 turns costs 394,000 tokens — four times the bill for twice the conversation. Nothing is broken; that is what a stateless endpoint means.

Prompt caching exists to blunt exactly this. If the unchanged prefix of a request — a system prompt, a retrieved document, a tool catalogue — is byte-identical to one processed recently, the server reuses the stored KV cache rather than recomputing it, and bills those tokens at a steep discount. That dictates the design: stable material first, volatile last, because one changed character near the top invalidates everything after it.

Streaming is the other structural difference. Because tokens are produced sequentially anyway, the server can emit them as they appear, which turns one latency number into two. Time to first token measures prefill and queueing; tokens per second measures decode speed. A 300-token answer at 50 tokens per second takes six seconds to finish, but an adult reads about 250 words a minute — near five tokens a second — so a stream ten times faster than the reader feels instant while the request is still running. Chat interfaces optimise the first number; batch jobs that parse finished JSON should ignore it.

Real-World Applications

The shape of one vendor's endpoint became an industry interface. Serving stacks such as vLLM, llama.cpp and Ollama, and aggregators such as OpenRouter and Together, all expose an OpenAI-compatible /v1/chat/completions route, which is why switching providers is usually a base URL and a key rather than a rewrite, and why a deployment can keep a self-hosted model and a hosted one behind one client.

Rate limits are the second place the abstraction becomes a hard constraint. Providers publish them in tokens per minute as well as requests per minute, because the scarce resource is accelerator time, not connection count. With a 200,000 token-per-minute allowance and an average call of 4,000 input plus 1,000 output tokens, you saturate at 40 requests per minute — 8% of a 500 request-per- minute ceiling you will therefore never reach. Capacity planning that counts requests plans the wrong number.

Tool calling is what turns a text endpoint into something that acts. The model calls nothing itself; it emits a structured request naming a function and its arguments, your code executes it, and the result is appended to the transcript and sent back. That loop is the whole of function calling and the substrate of every AI agent, and it carries a token cost people forget: schemas are part of the prompt, so eight tools at roughly 200 tokens of JSON schema each add 1,600 tokens to every request, and a twelve-step agent loop pays 19,200 before any useful work is counted. The Model Context Protocol standardises the tool side so a server written once works with any client, while Agent2Agent targets agent-to-agent traffic — the surviving standard there, after IBM's Agent Communication Protocol merged into it in August 2025.

Key Concepts

Two limits are routinely confused, and confusing them is how a reply goes missing:

• Context window versus output cap: how much you may send and how much the model may write back are separate ceilings, and hitting the second truncates a reply mid-sentence.
• Pinned model versions: model strings naming a dated snapshot hold behaviour still; floating aliases upgrade silently, invalidating whatever prompt engineering was tuned against them.

Challenges

Non-determinism is the awkward one, because it removes tools the rest of the industry relies on. You cannot cache a response by request hash if the response is legitimately different next time, cannot assert exact strings in tests, and cannot safely retry a request whose side effects already fired through a tool call. Even at temperature 0 reproducibility is not guaranteed: floating-point addition is not associative, so results shift with how your request was batched on the server.

Failure modes are unfamiliar too. A truncated reply arrives as a successful HTTP 200 with an unfinished sentence in it, so the only reliable signal is the stop-reason field, and a stream can fail after the connection succeeded. Cost per call is unbounded in a way ordinary APIs are not: a reasoning model can spend its entire output budget on internal thinking tokens and return nothing visible, fully billed.

Future Trends

The clearest direction is server-side state, which attacks the quadratic resend at its root: newer endpoints let a conversation or a cached prefix be referenced by id so the transcript stops crossing the wire every turn. Tiered service is becoming explicit alongside it — batch endpoints trading a long deadline for a discount, priority tiers buying latency — which is inference capacity sold by urgency the way cloud computing sells spot and reserved instances. Third is standardisation: tool interfaces are consolidating around MCP, which is what makes portable multi-agent systems plausible.

Code Example

Note the third message: it is text the model produced on the previous call, which your client stored and is now paying to send back as input.

POST /v1/chat/completions
{
  "model": "provider-model-2026-07-01",
  "stream": true,
  "max_tokens": 512,
  "temperature": 0.2,
  "messages": [
    { "role": "system",    "content": "You are a terse assistant." },
    { "role": "user",      "content": "What is a KV cache?" },
    { "role": "assistant", "content": "Stored key and value vectors..." },
    { "role": "user",      "content": "Why does it grow with context?" }
  ]
}

Pin the model to a dated snapshot, cap max_tokens so a runaway reply cannot run up the bill, and keep the stable prefix first so prompt caching can hit.