---
source: 'https://howaiworks.ai/glossary/conversational-ai'
section: glossary
title: Conversational AI
description: >-
  Conversational AI holds a multi-turn dialogue in natural language. Unlike an
  intent-based chatbot, it generates each reply rather than picking one.
tags:
  - Conversational AI
  - NLP
  - human-computer interaction
  - LLM
category: Natural Language Processing
datePublished: '2025-08-17'
lastUpdated: '2026-07-24'
---

# Conversational AI

> Conversational AI holds a multi-turn dialogue in natural language. Unlike an intent-based chatbot, it generates each reply rather than picking one.

## Definition

**Conversational AI** is software that holds a multi-turn dialogue with a person in ordinary
language — typed or spoken — keeping track of what was said earlier and, increasingly, taking
actions on that person's behalf. What separates it from the chatbot most people remember is a
single design decision: the old chatbot **selected** its reply from a list a human had written,
while a modern conversational system **generates** its reply.

The same generated-reply machinery serves a second and quite different design goal. A system built
to be *related to* rather than queried — reading tone and social context, and holding a position in
someone's life rather than answering their question — is [social AI](https://howaiworks.ai/glossary/social-ai), and the
failure modes there are about attachment and appropriateness rather than containment and handoff.

That is not a nuance, it is the whole trade. A support bot of the 2010s ran an intent classifier:
a fixed inventory of intents ("check balance", "reset password", "where is my parcel"), each with
a few dozen labelled example utterances and a scripted response. Anything outside the inventory
produced the sentence everyone has met — "I'm sorry, I didn't understand that." Bank of America's
Erica, launched in 2018, is the paradigm at full scale: by the time it passed two billion
interactions in April 2024, its data science team had made more than **50,000 updates** to its
language understanding. That number is the running cost of the approach. Every new thing the
assistant could say was a thing a person had to write.

A generative system has no inventory. It composes each reply from a
[large language model](https://howaiworks.ai/glossary/large-language-model), so the fallback disappears — and so does
the guarantee that came with it. The old system could only ever say what someone had authored and
approved; the new one can say anything, including a refund policy your company does not have. The
frustration and the safety property were the same property, and removing one removed the other.

|  | Intent-based chatbot | Generative conversational AI |
|---|---|---|
| Where the reply comes from | An authored list | Composed per turn by a model |
| Off-script input | "I didn't understand that" | A fluent answer, possibly invented |
| Adding a capability | New intent plus labelled utterances | New documents, tools and instructions |
| Safety model | The reply set is a whitelist | The reply set is unbounded; replies must be checked after they are produced |
| Typical failure | Visible and annoying | Invisible and confident |

This page is about the *conversation*: turn-taking, the state that spans turns, and what happens
when a turn goes wrong. When a system stops waiting for the next human message and starts
pursuing a goal across many steps of its own choosing, it has become an
[AI agent](https://howaiworks.ai/glossary/ai-agent), and the interesting questions change with it.

## How It Works

A production conversational system is a pipeline with the model in the middle of it, not a model
with a text box in front of it. Five things happen between a user's sentence and the reply.

**Understand.** The incoming message is read in the context of the transcript so far, which is
mostly a matter of resolving what the person is referring to: "cancel the second one" means
nothing without the list from three turns ago. Classical
[NLP](https://howaiworks.ai/glossary/natural-language-processing) called this coreference and slot filling and solved
it with dedicated components; a generative system gets it largely for free by being shown the
transcript, which is exactly why the transcript has to be resent.

**Retrieve.** The facts the business owns — the return policy, this customer's order, the current
opening hours — are fetched and placed in the prompt, the pattern known as
[retrieval-augmented generation](https://howaiworks.ai/glossary/retrieval-augmented-generation). This is where a
*correct* answer comes from. The model supplies fluency and structure; it must not be the source
of the facts.

**Act.** Anything with an effect in the world happens through a tool call — see
[function calling](https://howaiworks.ai/glossary/function-calling). It is worth splitting these in the design: read
actions (look up the order) can be taken freely, write actions (issue the refund) generally
should not be, because a confidently wrong write action is the failure that costs money.

**Generate.** The model produces the reply, conditioned on the retrieved facts, the tool results,
the transcript and a system prompt that carries the persona and the rules.

**Check.** A separate pass inspects the draft before it ships: policy classifiers, PII redaction,
sometimes a verification step that asks whether every claim in the draft is supported by a
retrieved document. This is the only stage that ever sees the sentence the customer will actually
read, which makes it the last place a bad one can be stopped.

The model is the small part. Two companies building on the same model produce entirely different
products, and the difference lives in the retrieval index and the tool layer — what the assistant
can look up and what it is permitted to do. That is also where the engineering time goes; swapping
the underlying model is a configuration change by comparison.

### State, memory, and what the assistant is allowed to forget

The model is stateless. It remembers nothing between calls, so the application resends the entire
conversation every turn and pays for all of it as input — which is why the input billed over a
chat grows with the *square* of the turn count. The [API](https://howaiworks.ai/glossary/api) page works that
calculation through; the consequence for conversation design is what matters here.

Take turns of roughly 150 tokens. By turn 20 each request carries about 3,000 tokens of history to
produce 150 new ones, so 95% of what crosses the wire is repetition, and the whole 20-turn session
has billed 150 × (1+2+…+20) = 31,500 input tokens to carry 3,000 tokens of actual conversation — a
10× multiplier that doubles again by turn 40. Long before cost bites, the transcript outgrows the
[context window](https://howaiworks.ai/glossary/context-window) and something has to give: summarise the early turns,
evict them, or store selected facts in a separate memory.

Choosing which is a product decision dressed as an engineering one, because it decides **what the
assistant is allowed to forget**. Drop the turn where the customer said the parcel arrived
damaged and the assistant will cheerfully ask again — the single most infuriating failure mode of
the genre, and one no benchmark measures. Summarisation trades this for a subtler risk: the
summary is itself generated, so a detail can be quietly rewritten rather than merely lost.

### The latency budget

Human conversation leaves roughly 200 milliseconds between turns, which is why voice assistants
are judged on time-to-first-audio rather than total render time — the
[text-to-speech](https://howaiworks.ai/glossary/text-to-speech) page covers that argument and the speech synthesis it
constrains. What conversational design owns is that the five stages above **share one budget and
add up**. A retrieval call at 100 ms, a tool call at 300 ms and 400 ms to the first generated
token puts you at 800 ms before a single word is spoken, and the safety check runs after that.
This is the concrete reason voice deployments retrieve less, call fewer tools and use smaller
models than their text equivalents: in a chat window a two-second pause is invisible, and in a
phone call it is a person saying "hello?"

## Real-World Applications

**Customer service, in both directions.** Klarna is the case study precisely because it has run
both experiments in public. In February 2024 the company reported that its assistant had handled
2.3 million conversations in its first month — two-thirds of its service chats, the work of about
700 full-time agents — and had cut average resolution time from 11 minutes to under 2. In May 2025
it began recruiting human agents again, with its CEO saying customers must always be able to reach
a person. Both halves are the lesson: the automation was real, and the reversal was not about the
model's quality but about what happened at the edge of its competence.

**High-volume banking assistants.** Erica has served more than 42 million clients and passed two
billion interactions by April 2024. It is a narrow, heavily constrained assistant, and it scaled
to that volume *because* its scope was constrained — a useful counterweight to the assumption
that generative always beats scripted. For a bounded set of high-frequency requests, an authored
answer that is always right beats a generated one that is usually right.

**Clinical documentation.** Systems that listen to a consultation and draft the note — see
[ambient clinical documentation](https://howaiworks.ai/glossary/ambient-clinical-documentation) — are conversational AI
with the roles inverted: the machine listens, and the "reply" is a document a professional signs.
The human review step is not a limitation of the current generation; it is the product.

**Voice front doors.** Phone menus built on [voice recognition](https://howaiworks.ai/glossary/voice-recognition) and
speech synthesis now route by asking what you want rather than by asking you to press 4. The
failure mode is specific: under the latency budget these systems retrieve less than their text
counterparts, so they answer from the model's own memory more often — which is where invented
policy comes from.

## Key Concepts

- **Turn**: one user message and the system's reply. It is the unit that latency, cost and quality are all measured in, and the unit that gets resent.
- **Containment (or deflection) rate**: the share of conversations closed without a person joining. The headline metric, and the easiest one to game.
- **Escalation**: handing the conversation to a human, with the transcript attached. Whether the transcript survives the handoff decides how the customer rates the whole interaction.
- **Grounding**: the property that every factual claim in a reply traces to a retrieved document rather than to the model's parameters. Ungrounded fluency is [hallucination](https://howaiworks.ai/glossary/hallucinations) with good manners.
- **Guardrail**: a check placed before or after the model. Only the output-side one sees the actual sentence, so only it can stop a specific sentence.

## Challenges

**Invented policy is a liability, not a bug report.** In *Moffatt v. Air Canada* (2024 BCCRT 149)
a tribunal held the airline responsible for a chatbot's description of a bereavement-fare refund
that its published policy did not offer, awarding C$650.88. The sum is trivial; the finding is
not. The tribunal rejected the argument that the chatbot was a separate entity, which is the
sentence to remember: whatever the assistant says, the company said. Grounding every claim in a
retrieved document is not a quality nicety, it is how you keep the assistant from writing new
policy in public.

**User text is not safely separable from instructions.** A conversational system reads its
transcript and its retrieved documents in the same channel it reads its own instructions, so
anything that can put text in front of the model can attempt to redirect it — the user, but also
whoever wrote the support ticket, the product review or the PDF the retrieval layer just pulled
in. Prompt injection has no clean fix, only mitigations: restrict what tools exist, require
confirmation on write actions, and treat retrieved content as hostile.

**There is no negative guarantee.** You cannot make a generative system *provably* never say a
given sentence. Instructions, fine-tuning and preference training all shift probabilities; none
of them sets one to zero. If a sentence must never be said — a price, a medical instruction, a
legal commitment — it has to be enforced outside the model, by a deterministic check on the output
or by serving that answer from a template. Designs that rely on the system prompt alone are
relying on persuasion.

**Evaluation has no ground truth, so every metric is a proxy — and every proxy is gameable.**
There is no single correct reply to "my order is late", so quality gets measured indirectly:
task completion, containment rate, escalation rate, customer satisfaction. Containment is the one
that ends up on the slide, and it counts conversations that ended without a human. An assistant
that simply never offers a handoff scores 100%.

The gaming does not have to be deliberate; tuning for the metric produces it on its own. Say
1,000 conversations arrive and 700 end without a person: 70% containment. If 200 of those
customers come back within a week, real containment is 500 in 1,000 — 50% — and those 200 cost an
automated conversation *plus* the human contact, plus a customer who now distrusts the channel.
The fix is not a better metric but a paired one: containment is only meaningful next to the
repeat-contact rate over the following 7 days, and satisfaction has to be measured on the
escalated conversations separately, because averaging them with the easy ones hides exactly the
population you are failing.

**The handoff decides satisfaction more than the model does.** When the assistant reaches the
edge of what it can do, what happens next determines how the entire conversation is remembered.
A handoff that carries the transcript, so the agent opens with "I see the parcel arrived damaged",
turns a failure into a decent experience. A handoff that drops it makes the customer repeat
everything, and the automated conversation becomes pure added waiting. This is unglamorous
integration work between the assistant and the ticketing system, and it is the part most
deployments underinvest in — Klarna's reversal was about this, not about the model.

## Future Trends

The clearest direction is removing stages from the pipeline. Speech-to-speech models take audio in
and emit audio out without passing through written text, cutting a round trip out of the latency
budget and preserving tone that transcription discards. Server-side conversation state and cached
prefixes attack the quadratic resend at its root, letting a transcript be referenced by id rather
than re-uploaded every turn — which changes the economics of a long conversation more than any
model improvement will.

The more consequential shift is in initiative. Today's systems act only when spoken to; the moment
one is allowed to do something between turns — chase a delayed shipment, notice a failed payment
and open the conversation itself — the design questions become those of an
[AI agent](https://howaiworks.ai/glossary/ai-agent), and so do the failure modes. The conversational interface stays;
what changes is who starts the turn.

## Frequently Asked Questions

### What's the difference between a chatbot and conversational AI?

The chatbot most people remember selected its reply from a list a human wrote, matching your sentence to one of a fixed set of intents and falling back to "I didn't understand that" for anything else. A modern conversational system generates the reply instead, so there is no fixed list — which removes the frustrating fallback and also removes the guarantee that the assistant can only say things someone approved.

### What is a containment or deflection rate, and why can it mislead?

It is the share of conversations that end without a person joining — the headline number vendors quote. It is trivially gamed: an assistant that never offers a handoff scores 100%. Read it alongside the repeat-contact rate, because a customer who gives up and calls back the next day was counted as a success.

### Does a conversational assistant actually reduce support cost?

The direct arithmetic is easy to clear. If a handled conversation costs b and a human contact costs H, the cost per inbound contact is b + (1 − r)·H at containment rate r, so you break even at r = b/H — around 5% when an automated conversation costs a twentieth of a human one. The real argument is about the customers in the 1 − r who waited for the assistant before reaching a person.

### How does a conversational AI remember what was said earlier?

It does not remember: the model is stateless, so the application resends the whole transcript on every turn. Once the conversation outgrows the context window, something has to be summarised or dropped, and deciding what the assistant is allowed to forget is a product decision — dropping the wrong turn produces the classic failure of asking a question the customer already answered.

### Can conversational AI handle multiple languages?

Yes — a multilingual LLM can hold the same conversation in dozens of languages and switch mid-conversation. The limiting factor is usually not the model but the retrieval layer: an assistant is only as multilingual as the policy documents it is allowed to quote.

### What are the main safety concerns with conversational AI?

Three that are specific to dialogue: the model can state a policy that does not exist and bind the company to it, user text can act as an instruction rather than as data (prompt injection), and there is no way to prove a generative system will never say a particular sentence. Conversation logs are also personal data by default, since people volunteer far more in chat than they would type into a form.

## Related

### Related terms

- [Natural Language Processing (NLP)](https://howaiworks.ai/glossary/natural-language-processing)
- [Large Language Model (LLM)](https://howaiworks.ai/glossary/large-language-model)
- [AI Agent](https://howaiworks.ai/glossary/ai-agent)
- [Voice Recognition](https://howaiworks.ai/glossary/voice-recognition)
- [Text Generation](https://howaiworks.ai/glossary/text-generation)
- [Multimodal AI](https://howaiworks.ai/glossary/multimodal-ai)

---

Source: https://howaiworks.ai/glossary/conversational-ai — HowAIWorks.ai