---
source: 'https://howaiworks.ai/glossary/api'
section: glossary
title: API
description: >-
  An API (Application Programming Interface) is a set of rules and protocols
  that allows different software applications to communicate and share data
  seamlessly.
tags:
  - API
  - Software Development
  - AI Agents
  - Tool Calling
category: Software Development
datePublished: '2026-01-30'
lastUpdated: '2026-07-24'
---

# API

> An API (Application Programming Interface) is a set of rules and protocols that allows different software applications to communicate and share data seamlessly.

## Definition

An **API (Application Programming Interface)** is the published contract by which one program calls
another. In AI that contract is almost always an HTTPS endpoint you send a JSON body to: you post
some text, a machine elsewhere performs a few hundred billion multiplications, and text comes back.
Almost nobody runs a frontier [large language model](https://howaiworks.ai/glossary/large-language-model) themselves, so
the API *is* the product.

What earns a model API its own entry is that it breaks three assumptions every developer brings from
ordinary web services. It is billed by the [token](https://howaiworks.ai/glossary/token) rather than by the request, so
two calls to the same URL can differ in cost a thousandfold. It is **stateless**: the model
remembers nothing between calls, so a "conversation" is your own client resending the entire
transcript every turn. And the same request sent twice can legitimately return **different text**,
which quietly invalidates response caching, safe retries and most test assertions. Miss any one of
those and you ship a bug or a surprise invoice.

## How It Works

A request carries a model identifier, a list of messages with roles, and sampling settings such as
[temperature](https://howaiworks.ai/glossary/temperature). The server turns the messages into token ids and runs the
model, and that work splits into two phases with very different physics. **Prefill** processes the
whole prompt in one parallel pass, because all of its tokens are known at once. **Decode** then
produces the reply one token at a time, each token needing its own full pass through the model,
because token 400 cannot be computed before token 399 exists. Decode is limited by how fast weights
can be read from memory rather than by arithmetic — a 70-billion parameter model in 8-bit precision
means reading roughly 70 GB per token, which on a 3.35 TB/s accelerator caps a single stream near 47
tokens per second (see [the memory wall](https://howaiworks.ai/glossary/memory-wall)). That asymmetry, not vendor margin,
is why output tokens cost several times what input tokens cost.

Statelessness is the property that costs people real money, so do the arithmetic. Suppose each turn
of a chat adds 500 tokens — a 100-token question and a 400-token answer. On turn *n* your client
sends everything said so far plus the new question: 500 × (n − 1) + 100 tokens. Across 20 turns that
totals 500 × 190 + 2,000 = **97,000 input tokens billed to hold a conversation whose full text is
only 10,000 tokens**, so you paid for the transcript nearly ten times over. Because the sum is
n(n−1)/2, doubling the chat to 40 turns costs 394,000 tokens — **four times the bill for twice the
conversation**. Nothing is broken; that is what a stateless endpoint means.

Prompt caching exists to blunt exactly this. If the unchanged prefix of a request — a system prompt,
a retrieved document, a tool catalogue — is byte-identical to one processed recently, the server
reuses the stored [KV cache](https://howaiworks.ai/glossary/kv-cache) rather than recomputing it, and bills those tokens
at a steep discount. That dictates the design: stable material first, volatile last, because one
changed character near the top invalidates everything after it.

Streaming is the other structural difference. Because tokens are produced sequentially anyway, the
server can emit them as they appear, which turns one latency number into two. **Time to first
token** measures prefill and queueing; **tokens per second** measures decode speed. A 300-token
answer at 50 tokens per second takes six seconds to finish, but an adult reads about 250 words a
minute — near five tokens a second — so a stream ten times faster than the reader feels instant
while the request is still running. Chat interfaces optimise the first number; batch jobs that parse
finished JSON should ignore it.

## Real-World Applications

The shape of one vendor's endpoint became an industry interface. Serving stacks such as vLLM,
llama.cpp and Ollama, and aggregators such as OpenRouter and Together, all expose an
OpenAI-compatible `/v1/chat/completions` route, which is why switching providers is usually a base
URL and a key rather than a rewrite, and why a [deployment](https://howaiworks.ai/glossary/model-deployment) can keep a
self-hosted model and a hosted one behind one client.

Rate limits are the second place the abstraction becomes a hard constraint. Providers publish them
in **tokens per minute as well as requests per minute**, because the scarce resource is accelerator
time, not connection count. With a 200,000 token-per-minute allowance and an average call of 4,000
input plus 1,000 output tokens, you saturate at 40 requests per minute — 8% of a 500 request-per-
minute ceiling you will therefore never reach. Capacity planning that counts requests plans the
wrong number.

Tool calling is what turns a text endpoint into something that acts. The model calls nothing itself;
it emits a structured request naming a function and its arguments, your code executes it, and the
result is appended to the transcript and sent back. That loop is the whole of
[function calling](https://howaiworks.ai/glossary/function-calling) and the substrate of every
[AI agent](https://howaiworks.ai/glossary/ai-agent), and it carries a token cost people forget: schemas are part of the
prompt, so eight tools at roughly 200 tokens of JSON schema each add 1,600 tokens to *every*
request, and a twelve-step agent loop pays 19,200 before any useful work is counted. The
[Model Context Protocol](https://howaiworks.ai/glossary/model-context-protocol) standardises the tool side so a server
written once works with any client, while [Agent2Agent](https://howaiworks.ai/glossary/agent2agent-protocol) targets
agent-to-agent traffic — the surviving standard there, after IBM's
[Agent Communication Protocol](https://howaiworks.ai/glossary/agent-communication-protocol) merged into it in August 2025.

## Key Concepts

Two limits are routinely confused, and confusing them is how a reply goes missing:

- **Context window versus output cap**: how much you may send and how much the model may write back
  are separate ceilings, and hitting the second truncates a reply mid-sentence.
- **Pinned model versions**: `model` strings naming a dated snapshot hold behaviour still; floating
  aliases upgrade silently, invalidating whatever
  [prompt engineering](https://howaiworks.ai/glossary/prompt-engineering) was tuned against them.

## Challenges

Non-determinism is the awkward one, because it removes tools the rest of the industry relies on. You
cannot cache a response by request hash if the response is legitimately different next time, cannot
assert exact strings in tests, and cannot safely retry a request whose side effects already fired
through a tool call. Even at temperature 0 reproducibility is not guaranteed: floating-point
addition is not associative, so results shift with how your request was batched on the server.

Failure modes are unfamiliar too. A truncated reply arrives as a successful HTTP 200 with an
unfinished sentence in it, so the only reliable signal is the stop-reason field, and a stream can
fail *after* the connection succeeded. Cost per call is unbounded in a way ordinary APIs are not: a
reasoning model can spend its entire output budget on internal thinking tokens and return nothing
visible, fully billed.

## Future Trends

The clearest direction is server-side state, which attacks the quadratic resend at its root: newer
endpoints let a conversation or a cached prefix be referenced by id so the transcript stops crossing
the wire every turn. Tiered service is becoming explicit alongside it — batch endpoints trading a
long deadline for a discount, priority tiers buying latency — which is
[inference](https://howaiworks.ai/glossary/inference) capacity sold by urgency the way [cloud
computing](https://howaiworks.ai/glossary/cloud-computing) sells spot and reserved instances. Third is standardisation:
tool interfaces are consolidating around MCP, which is what makes portable
[multi-agent systems](https://howaiworks.ai/glossary/multi-agent-systems) plausible.

## Code Example

Note the third message: it is text the *model* produced on the previous call, which your client
stored and is now paying to send back as input.

```json
POST /v1/chat/completions
{
  "model": "provider-model-2026-07-01",
  "stream": true,
  "max_tokens": 512,
  "temperature": 0.2,
  "messages": [
    { "role": "system",    "content": "You are a terse assistant." },
    { "role": "user",      "content": "What is a KV cache?" },
    { "role": "assistant", "content": "Stored key and value vectors..." },
    { "role": "user",      "content": "Why does it grow with context?" }
  ]
}
```

Pin the model to a dated snapshot, cap `max_tokens` so a runaway reply cannot run up the bill, and
keep the stable prefix first so prompt caching can hit.

## Frequently Asked Questions

### Why are APIs important for AI?

APIs allow AI models to connect to the real world. Through APIs, an AI can search the web, send emails, access databases, and control other software applications, turning a 'chatbox' into an 'autonomous agent'.

### What is an API endpoint?

An endpoint is a specific URL where an API receives requests. For example, OpenAI's API has an endpoint for generating text and another for generating images.

### Why does a long chat cost more per message than a short one?

Because a model API is stateless. The server remembers nothing between calls, so your client resends the entire conversation every turn and is billed for all of it as input. Over a 20-turn chat the input billed grows with the square of the turn count, not linearly.

### Why does the same API request return a different answer each time?

The model produces a probability distribution over the next token and one is sampled from it, so the reply is drawn rather than looked up. Setting temperature to 0 makes it near-deterministic but not exactly reproducible, because floating-point results shift with how requests are batched on the server.

## Related

### Related terms

- [AI Agent](https://howaiworks.ai/glossary/ai-agent)
- [Model Context Protocol (MCP)](https://howaiworks.ai/glossary/model-context-protocol)
- [Function Calling (Tool Calling)](https://howaiworks.ai/glossary/function-calling)
- [Cloud Computing](https://howaiworks.ai/glossary/cloud-computing)

---

Source: https://howaiworks.ai/glossary/api — HowAIWorks.ai