Definition
Conversational AI is software that holds a multi-turn dialogue with a person in ordinary language — typed or spoken — keeping track of what was said earlier and, increasingly, taking actions on that person's behalf. What separates it from the chatbot most people remember is a single design decision: the old chatbot selected its reply from a list a human had written, while a modern conversational system generates its reply.
The same generated-reply machinery serves a second and quite different design goal. A system built to be related to rather than queried — reading tone and social context, and holding a position in someone's life rather than answering their question — is social AI, and the failure modes there are about attachment and appropriateness rather than containment and handoff.
That is not a nuance, it is the whole trade. A support bot of the 2010s ran an intent classifier: a fixed inventory of intents ("check balance", "reset password", "where is my parcel"), each with a few dozen labelled example utterances and a scripted response. Anything outside the inventory produced the sentence everyone has met — "I'm sorry, I didn't understand that." Bank of America's Erica, launched in 2018, is the paradigm at full scale: by the time it passed two billion interactions in April 2024, its data science team had made more than 50,000 updates to its language understanding. That number is the running cost of the approach. Every new thing the assistant could say was a thing a person had to write.
A generative system has no inventory. It composes each reply from a large language model, so the fallback disappears — and so does the guarantee that came with it. The old system could only ever say what someone had authored and approved; the new one can say anything, including a refund policy your company does not have. The frustration and the safety property were the same property, and removing one removed the other.
| Intent-based chatbot | Generative conversational AI | |
|---|---|---|
| Where the reply comes from | An authored list | Composed per turn by a model |
| Off-script input | "I didn't understand that" | A fluent answer, possibly invented |
| Adding a capability | New intent plus labelled utterances | New documents, tools and instructions |
| Safety model | The reply set is a whitelist | The reply set is unbounded; replies must be checked after they are produced |
| Typical failure | Visible and annoying | Invisible and confident |
This page is about the conversation: turn-taking, the state that spans turns, and what happens when a turn goes wrong. When a system stops waiting for the next human message and starts pursuing a goal across many steps of its own choosing, it has become an AI agent, and the interesting questions change with it.
How It Works
A production conversational system is a pipeline with the model in the middle of it, not a model with a text box in front of it. Five things happen between a user's sentence and the reply.
Understand. The incoming message is read in the context of the transcript so far, which is mostly a matter of resolving what the person is referring to: "cancel the second one" means nothing without the list from three turns ago. Classical NLP called this coreference and slot filling and solved it with dedicated components; a generative system gets it largely for free by being shown the transcript, which is exactly why the transcript has to be resent.
Retrieve. The facts the business owns — the return policy, this customer's order, the current opening hours — are fetched and placed in the prompt, the pattern known as retrieval-augmented generation. This is where a correct answer comes from. The model supplies fluency and structure; it must not be the source of the facts.
Act. Anything with an effect in the world happens through a tool call — see function calling. It is worth splitting these in the design: read actions (look up the order) can be taken freely, write actions (issue the refund) generally should not be, because a confidently wrong write action is the failure that costs money.
Generate. The model produces the reply, conditioned on the retrieved facts, the tool results, the transcript and a system prompt that carries the persona and the rules.
Check. A separate pass inspects the draft before it ships: policy classifiers, PII redaction, sometimes a verification step that asks whether every claim in the draft is supported by a retrieved document. This is the only stage that ever sees the sentence the customer will actually read, which makes it the last place a bad one can be stopped.
The model is the small part. Two companies building on the same model produce entirely different products, and the difference lives in the retrieval index and the tool layer — what the assistant can look up and what it is permitted to do. That is also where the engineering time goes; swapping the underlying model is a configuration change by comparison.
State, memory, and what the assistant is allowed to forget
The model is stateless. It remembers nothing between calls, so the application resends the entire conversation every turn and pays for all of it as input — which is why the input billed over a chat grows with the square of the turn count. The API page works that calculation through; the consequence for conversation design is what matters here.
Take turns of roughly 150 tokens. By turn 20 each request carries about 3,000 tokens of history to produce 150 new ones, so 95% of what crosses the wire is repetition, and the whole 20-turn session has billed 150 × (1+2+…+20) = 31,500 input tokens to carry 3,000 tokens of actual conversation — a 10× multiplier that doubles again by turn 40. Long before cost bites, the transcript outgrows the context window and something has to give: summarise the early turns, evict them, or store selected facts in a separate memory.
Choosing which is a product decision dressed as an engineering one, because it decides what the assistant is allowed to forget. Drop the turn where the customer said the parcel arrived damaged and the assistant will cheerfully ask again — the single most infuriating failure mode of the genre, and one no benchmark measures. Summarisation trades this for a subtler risk: the summary is itself generated, so a detail can be quietly rewritten rather than merely lost.
The latency budget
Human conversation leaves roughly 200 milliseconds between turns, which is why voice assistants are judged on time-to-first-audio rather than total render time — the text-to-speech page covers that argument and the speech synthesis it constrains. What conversational design owns is that the five stages above share one budget and add up. A retrieval call at 100 ms, a tool call at 300 ms and 400 ms to the first generated token puts you at 800 ms before a single word is spoken, and the safety check runs after that. This is the concrete reason voice deployments retrieve less, call fewer tools and use smaller models than their text equivalents: in a chat window a two-second pause is invisible, and in a phone call it is a person saying "hello?"
Real-World Applications
Customer service, in both directions. Klarna is the case study precisely because it has run both experiments in public. In February 2024 the company reported that its assistant had handled 2.3 million conversations in its first month — two-thirds of its service chats, the work of about 700 full-time agents — and had cut average resolution time from 11 minutes to under 2. In May 2025 it began recruiting human agents again, with its CEO saying customers must always be able to reach a person. Both halves are the lesson: the automation was real, and the reversal was not about the model's quality but about what happened at the edge of its competence.
High-volume banking assistants. Erica has served more than 42 million clients and passed two billion interactions by April 2024. It is a narrow, heavily constrained assistant, and it scaled to that volume because its scope was constrained — a useful counterweight to the assumption that generative always beats scripted. For a bounded set of high-frequency requests, an authored answer that is always right beats a generated one that is usually right.
Clinical documentation. Systems that listen to a consultation and draft the note — see ambient clinical documentation — are conversational AI with the roles inverted: the machine listens, and the "reply" is a document a professional signs. The human review step is not a limitation of the current generation; it is the product.
Voice front doors. Phone menus built on voice recognition and speech synthesis now route by asking what you want rather than by asking you to press 4. The failure mode is specific: under the latency budget these systems retrieve less than their text counterparts, so they answer from the model's own memory more often — which is where invented policy comes from.
Key Concepts
- Turn: one user message and the system's reply. It is the unit that latency, cost and quality are all measured in, and the unit that gets resent.
- Containment (or deflection) rate: the share of conversations closed without a person joining. The headline metric, and the easiest one to game.
- Escalation: handing the conversation to a human, with the transcript attached. Whether the transcript survives the handoff decides how the customer rates the whole interaction.
- Grounding: the property that every factual claim in a reply traces to a retrieved document rather than to the model's parameters. Ungrounded fluency is hallucination with good manners.
- Guardrail: a check placed before or after the model. Only the output-side one sees the actual sentence, so only it can stop a specific sentence.
Challenges
Invented policy is a liability, not a bug report. In Moffatt v. Air Canada (2024 BCCRT 149) a tribunal held the airline responsible for a chatbot's description of a bereavement-fare refund that its published policy did not offer, awarding C$650.88. The sum is trivial; the finding is not. The tribunal rejected the argument that the chatbot was a separate entity, which is the sentence to remember: whatever the assistant says, the company said. Grounding every claim in a retrieved document is not a quality nicety, it is how you keep the assistant from writing new policy in public.
User text is not safely separable from instructions. A conversational system reads its transcript and its retrieved documents in the same channel it reads its own instructions, so anything that can put text in front of the model can attempt to redirect it — the user, but also whoever wrote the support ticket, the product review or the PDF the retrieval layer just pulled in. Prompt injection has no clean fix, only mitigations: restrict what tools exist, require confirmation on write actions, and treat retrieved content as hostile.
There is no negative guarantee. You cannot make a generative system provably never say a given sentence. Instructions, fine-tuning and preference training all shift probabilities; none of them sets one to zero. If a sentence must never be said — a price, a medical instruction, a legal commitment — it has to be enforced outside the model, by a deterministic check on the output or by serving that answer from a template. Designs that rely on the system prompt alone are relying on persuasion.
Evaluation has no ground truth, so every metric is a proxy — and every proxy is gameable. There is no single correct reply to "my order is late", so quality gets measured indirectly: task completion, containment rate, escalation rate, customer satisfaction. Containment is the one that ends up on the slide, and it counts conversations that ended without a human. An assistant that simply never offers a handoff scores 100%.
The gaming does not have to be deliberate; tuning for the metric produces it on its own. Say 1,000 conversations arrive and 700 end without a person: 70% containment. If 200 of those customers come back within a week, real containment is 500 in 1,000 — 50% — and those 200 cost an automated conversation plus the human contact, plus a customer who now distrusts the channel. The fix is not a better metric but a paired one: containment is only meaningful next to the repeat-contact rate over the following 7 days, and satisfaction has to be measured on the escalated conversations separately, because averaging them with the easy ones hides exactly the population you are failing.
The handoff decides satisfaction more than the model does. When the assistant reaches the edge of what it can do, what happens next determines how the entire conversation is remembered. A handoff that carries the transcript, so the agent opens with "I see the parcel arrived damaged", turns a failure into a decent experience. A handoff that drops it makes the customer repeat everything, and the automated conversation becomes pure added waiting. This is unglamorous integration work between the assistant and the ticketing system, and it is the part most deployments underinvest in — Klarna's reversal was about this, not about the model.
Future Trends
The clearest direction is removing stages from the pipeline. Speech-to-speech models take audio in and emit audio out without passing through written text, cutting a round trip out of the latency budget and preserving tone that transcription discards. Server-side conversation state and cached prefixes attack the quadratic resend at its root, letting a transcript be referenced by id rather than re-uploaded every turn — which changes the economics of a long conversation more than any model improvement will.
The more consequential shift is in initiative. Today's systems act only when spoken to; the moment one is allowed to do something between turns — chase a delayed shipment, notice a failed payment and open the conversation itself — the design questions become those of an AI agent, and so do the failure modes. The conversational interface stays; what changes is who starts the turn.