---
source: 'https://howaiworks.ai/glossary/natural-language-processing'
section: glossary
title: Natural Language Processing (NLP)
description: >-
  NLP is the field; large language models are now the answer to most of it. What
  sixty years of work was for, and which parts are still unsolved.
tags:
  - NLP
  - LLM
  - Language Model
  - Multilingual AI
  - Benchmarks
category: Natural Language Processing
datePublished: '2025-07-16'
lastUpdated: '2026-07-22'
---

# Natural Language Processing (NLP)

> NLP is the field; large language models are now the answer to most of it. What sixty years of work was for, and which parts are still unsolved.

## Definition

Natural language processing is the field of getting computers to do useful things with human
language — and since roughly 2020, one kind of model has been the answer to most of it. That is
the first thing to settle, because it is the live confusion: **NLP is a set of problems, a
[large language model](https://howaiworks.ai/glossary/large-language-model) is the current answer to them.** Asking
whether NLP and LLMs are the same thing is like asking whether medicine and antibiotics are the
same thing. Translation, summarisation, entity extraction and question answering are NLP tasks and
were NLP tasks decades before a transformer existed; an LLM is simply the technique that now does
them.

Here is what those problems actually look like, on numbers. Take the sentence Kenneth Church and
Ramesh Patil put in the title of their 1982 paper: *Put the block in the box on the table.* It has
two readings — put the block **into** the box that is on the table, or put the block **that is in
the box** onto the table. Nothing in the words decides it; you decide by knowing things about
blocks, boxes and tables. Add a third prepositional phrase and there are 5 readings, a fourth gives
14, a fifth gives 42. The counts are the Catalan numbers, and the paper reports that their parser
found exactly that many parses for real sentences all the way out to nine prepositional phrases —
**4,862 structurally valid analyses** of a sentence a person reads once without noticing any
ambiguity at all.

That gap — between what a sentence formally permits and what a reader effortlessly picks — is the
whole of NLP in one line. Sixty years of the field is a sequence of attempts to close it. Hand
written rules closed it for a paragraph at a time and then broke. Annotated corpora and one
statistical model per task closed it further, at enormous human cost. What finally moved it was
scale: GPT-3 (Brown et al., 2020) noted that the prevailing method "still requires task-specific
fine-tuning datasets of thousands or tens of thousands of examples", then showed a single
175-billion-parameter model doing many of those same tasks from a handful of examples typed into
the prompt, with no gradient updates at all.

The tasks did not disappear. The separate models mostly did — which is exactly why a 2026 job
advert for an NLP engineer still lists named-entity recognition, part-of-speech tagging, parsing
and sentiment analysis as if they were separate skills. They were, for thirty years. Knowing which
of them still needs its own model is now a large part of the job.

And here is what breaks if you get the distinction wrong: you start reading benchmark scores as
capability claims. A model that tops an English leaderboard has demonstrated something about that
leaderboard, in that language. It has not demonstrated that it will parse your clinical notes, or
that it works in Yoruba, or that the metric you scored it with measures what you care about. Every
one of those has a measured counter-example further down this page.

## How It Works

### Three problems that turned out to be hard for non-obvious reasons

**Ambiguity is combinatorial, not occasional.** The Catalan explosion above is the structural case,
and lexical ambiguity multiplies on top of it: every word with several senses forks the reading
again. The counter-intuitive part is not that ambiguity exists but that people never see it. A
system that enumerates possible analyses does the linguistically correct thing and drowns; a reader
who does the linguistically incorrect thing — commits instantly to one reading using world
knowledge — is right almost always.

**Reference has no local answer.** Consider the pair Levesque, Davis and Morgenstern built their
2012 challenge around, adapted from Terry Winograd's 1972 work: *The town councillors refused to
give the angry demonstrators a permit because they **feared** violence* versus the same sentence
with **advocated**. Change one word and the pronoun *they* flips from the councillors to the
demonstrators. No amount of grammatical analysis of the sentence helps, and no co-occurrence
statistic over a corpus helps either, because the two sentences are nearly identical strings. You
need a model of who fears what.

**Context reaches outside the sentence entirely.** Google's own illustration when it deployed BERT
in Search in October 2019 was the query *2019 brazil traveler to usa need a visa*. The word "to"
carries the whole meaning — who is travelling where — and Google's blog post is blunt that its
earlier algorithms "wouldn't understand the importance of this connection, and we returned results
about U.S. citizens traveling to Brazil". The information that resolves the query is a function
word that a keyword system throws away.

### The task list, and what happened to it

The vocabulary a newcomer meets — tokenisation, part-of-speech tagging, syntactic parsing,
named-entity recognition, coreference resolution, sentiment classification, machine translation,
summarisation, question answering — is a list of *tasks*, not a taxonomy of the field. Each was
once a research community with its own annotated dataset, its own model architecture and its own
leaderboard. A 2015 production pipeline genuinely ran several of them in sequence: tokenise, tag,
parse, then extract.

Most of that pipeline is now one prompt. What survives as separate machinery is the parts that are
either cheaper or more controllable done separately: [tokenization](https://howaiworks.ai/glossary/tokenization) still
happens, because it is how text becomes numbers at all; retrieval and ranking still run on small
[embedding](https://howaiworks.ai/glossary/embedding) models because you cannot afford to call a frontier model once per
document in an index; and high-volume classification — spam, moderation, routing — often stays with
a small fine-tuned encoder for the same reason.

### Three eras, and what each one cost

**Rules, written by hand.** Joseph Weizenbaum's ELIZA (1966) is the honest specimen, because its
author documented exactly how little was inside it. Input was matched against keywords, which
triggered decomposition rules, which fired reassembly rules to build a reply. Weizenbaum's own
assessment is the part worth keeping: the program "maintains the illusion of understanding with so
little machinery", and "throws away each of its inputs" except the fragments it transforms. His
test for the real thing is still a good one — "the crucial test of understanding... is not the
subject's ability to continue a conversation, but to draw valid conclusions from what he is being
told."

**Annotate a corpus, train one model per task.** The Penn Treebank is the monument of this era.
Marcus, Marcinkiewicz and Santorini report that its first phase (1989–1992) produced **4,885,798
tokens tagged for part of speech and 2,881,188 tokens annotated with syntactic structure**. The
cost is the interesting number. Annotators bracketing sentences entirely by hand managed about
**375 words per hour after three weeks**; correcting a parser's output instead, they reached
**750 to over 1,000 words per hour after three or four months' experience**. At 750 words an hour,
the parsed portion alone represents roughly **3,800 annotator-hours** — about two person-years of
skilled linguistic labour for under three million words.

**Pretrain on unlabelled text, then prompt.** Llama 3's flagship was trained on 15.6 trillion
tokens. At the 3.94 characters per token its own paper measures, that is about 61 trillion
characters, or on the order of **10 trillion words — roughly 3.5 million times the size of the
Penn Treebank's parsed portion**. Hand-annotating a corpus that size at the treebank's best
sustained rate would take about **seven million person-years**. That arithmetic is the entire
argument for [self-supervised learning](https://howaiworks.ai/glossary/self-supervised-learning): the supervision has to
come from the text predicting itself, because there is no version of the world in which it comes
from people.

### What the model is doing, in one paragraph

Text is split into subword units by a [tokenizer](https://howaiworks.ai/glossary/tokenization), each unit becomes a
vector, and a stack of [transformer](https://howaiworks.ai/glossary/transformer) layers uses
[attention](https://howaiworks.ai/glossary/attention-mechanism) to let every position draw on every earlier one — which
is the mechanism that finally handled reference and long-range context without anyone writing a
rule for them. The model then predicts one token at a time. The
[large language model](https://howaiworks.ai/glossary/large-language-model) page covers that loop, its costs and where
its knowledge lives; the point for NLP is narrower and stranger. A single objective — predict the
next token — turned out to subsume a field's worth of separately-defined tasks, and nobody designed
it that way.

## Real-World Applications

**Search.** Google announced BERT in Search on 25 October 2019 with the claim that it "will help
Search better understand one in 10 searches in the U.S. in English". That is a deployment number,
not a benchmark: a transformer language model placed between a billion daily queries and their
results. [Semantic search](https://howaiworks.ai/glossary/semantic-search) and
[retrieval-augmented generation](https://howaiworks.ai/glossary/retrieval-augmented-generation) are the same idea grown
up — the query is understood by a model, and the documents are found by comparing vectors rather
than matching words.

**Machine translation.** Google's 2016 GNMT paper is the cleanest before-and-after in the field's
history: replacing the phrase-based production system with a neural one "reduces translation errors
by an average of 60%" on a human side-by-side evaluation. Their production table puts GNMT at 5.428
against a human translator's 5.504 on English-to-Spanish, and 4.594 against 4.987 on
English-to-Chinese, on a six-point scale. [DeepL](https://howaiworks.ai/ai-tools/deepl) built a business on the same
architecture. Note what the two pairs of numbers already show: the residual gap to human quality on
English-to-Chinese is more than five times the gap on English-to-Spanish, in the same system on the
same day.

**General assistants.** [ChatGPT](https://howaiworks.ai/ai-tools/chatgpt), [Claude](https://howaiworks.ai/ai-tools/claude) and
[Gemini](https://howaiworks.ai/ai-tools/google-gemini) are the visible face of the collapse described above: one model,
prompted differently, doing summarisation, translation, extraction and classification without a
task-specific model behind any of them. This is also why "NLP product" stopped being a useful
category — a legal-summary tool and a customer-support router are now the same model with different
context.

**Speech and clinical documentation.** [Voice recognition](https://howaiworks.ai/glossary/voice-recognition) and
[text-to-speech](https://howaiworks.ai/glossary/text-to-speech) converged on the same architecture from the audio side,
which is what makes [ambient clinical documentation](https://howaiworks.ai/glossary/ambient-clinical-documentation)
possible: a microphone in the consulting room, a transcript, and a structured note generated from
it. The NLP problem there is not transcription but knowing which parts of a conversation belong in
a medical record.

**The invisible majority.** Spam filtering, content moderation, support-ticket routing, resume
screening and product-review analysis run at volumes where calling a frontier model per item is
absurd. These mostly still use small fine-tuned classifiers, and they are the largest deployment of
NLP by item count. A model that reads a document once and emits a label does a fraction of the work
of one that reads a prompt template plus the document and then writes a sentence.

## Challenges

**Automatic metrics reward surface overlap, and everyone knows it.** BLEU (Papineni et al., 2002)
scores a translation by counting n-gram matches against references. Run it on *the dog bit the man*
scored against the reference *the dog bit the man* reversed — that is, the candidate *the man bit
the dog* — and modified unigram precision is **5/5**, a perfect score for a sentence that says the
opposite thing. Bigram precision is 3/4 and it only collapses at the trigram level. The BLEU
authors were candid about the failure mode in the original paper: a system producing the fluent
"East Asian economy" is "penalized heavily on the longer n-gram precisions if all the references
happen to read 'economy of East Asia'."

**Optimising the metric is not the same as improving the system, and there is a measurement of
it.** The GNMT team refined their models with reinforcement learning against BLEU directly. BLEU
rose from 40.35 to 41.16 — **+0.81 points** — while the human side-by-side score went from 4.46 to
**4.44**, fractionally down. Their own conclusion is that "even though RL refinement can achieve
better BLEU scores, it barely improves the human impression of the translation quality", and they
stopped using it in production experiments. When you optimise a proxy hard enough, you get the
proxy.

**Benchmarks saturate faster than capabilities improve.** GLUE, nine language-understanding tasks,
was published in April 2018. By early July 2019 the state of the art stood at **88.4 against a
non-expert human baseline of 87.1**, exceeding it on four of the nine tasks — so its authors
shipped SuperGLUE, on which the human estimate was 89.8 and a strong BERT baseline managed 71.5.
The lesson is not that the benchmark was bad. It is that a [benchmark](https://howaiworks.ai/glossary/benchmark) measures
a test, that the gap between test and capability is invisible from the score, and that fifteen
months is roughly how long a hard NLP benchmark now lasts.

**English results do not transfer, and the resource gap is not close to uniform.** Joshi et al.
classified **2,485 languages** by how much labelled and unlabelled data exists for each. Seven
languages — English, Spanish, German, Japanese, French among them — make up **0.28% of the total**
and take the benefit of every advance. **2,191 languages, 88.17% of the total and about a billion
speakers, sit in the bottom class with effectively no representation in any resource they measured.**
A frontier model's multilingual claims are downstream of that distribution, not independent of it.

**The model cannot see characters.** Counting letters, detecting rhyme, spotting a typo and
reasoning about spelling all fail for a reason that has nothing to do with language understanding:
the units the model receives are subword ids, not letters. This is a
[tokenization](https://howaiworks.ai/glossary/tokenization) property rather than an NLP one, but it produces some of the
most confidently-cited "proof that models don't understand language", which is why it is worth
knowing where the failure actually lives.

**Fluent output is not calibrated output.** Nothing in next-token prediction produces a confidence
signal a reader can use, so a wrong extraction reads exactly like a right one. For classification
at volume this is the strongest remaining argument for a small supervised model over a prompt: a
fine-tuned classifier gives you a probability you can threshold, and a generated sentence gives you
prose. [Hallucination](https://howaiworks.ai/glossary/hallucinations) is the same property showing up in longer form.

## Future Trends

**Evaluation is the field's live research problem.** With benchmarks saturating in about a year,
the work has moved to adversarially-constructed test sets, held-out private evaluations, and
model-graded rubrics — each of which has its own failure mode, and none of which has the clean
reproducibility that made BLEU and GLUE so useful. Expect the credible claims to come with a
described evaluation set rather than a single number.

**Low-resource languages are the largest unbanked value in the field.** The route is no longer
"collect a parallel corpus for each pair" but transfer: pretrain multilingually, then rely on
representations shared across languages to carry a task learned in one into another. It works
unevenly, and the two levers that matter most are unglamorous — the
[tokenizer's](https://howaiworks.ai/glossary/tokenization) allocation of vocabulary across writing systems, and the
amount of unlabelled text in the language at all. The 2,191 languages in Joshi's bottom class do
not have Wikipedia editions to pretrain on.

**Meaning grounded in something other than text.** The tasks NLP has not solved cluster around
reference to things outside the document: pronouns across a hundred-page contract, a claim that
must be checked against a database, an instruction whose correctness depends on the state of the
world. [Multimodal](https://howaiworks.ai/glossary/multimodal-ai) training and tool use are both attempts to attach
language to a referent rather than to more language. This is the oldest open problem in the field,
and it is the one a larger pretraining run has so far moved least.

**Small models coming back for narrow work.** Distillation plus fine-tuning keeps producing task
models that match a frontier model on one task at a fraction of the cost per item, which matters
precisely where volume is high and the task is stable. The pendulum swung all the way to "one
general model for everything" and is settling somewhere sensible: general models for the long tail,
specialised ones for the head.

## Code Example

The ambiguity explosion in the Definition is worth reproducing, because it is a fact about English
structure rather than about any model. The number of ways to bracket *n* prepositional phrases is
the number of binary trees over *n* + 1 leaves:

```python
from functools import lru_cache

@lru_cache(None)
def bracketings(leaves):
    """Ways to fully parenthesize a sequence of `leaves` items."""
    if leaves == 1:
        return 1
    return sum(bracketings(i) * bracketings(leaves - i) for i in range(1, leaves))

for pps in range(1, 10):
    print(f"{pps} PPs -> {bracketings(pps + 1):>5} parses")
```

Real output:

```
1 PPs ->     1 parses
2 PPs ->     2 parses
3 PPs ->     5 parses
4 PPs ->    14 parses
5 PPs ->    42 parses
6 PPs ->   132 parses
7 PPs ->   429 parses
8 PPs ->  1430 parses
9 PPs ->  4862 parses
```

Those are the Catalan numbers, and the 4,862 at nine phrases is the figure Church and Patil report
their parser actually returning. The second block computes BLEU's modified n-gram precision on the
word-order example from the Challenges section:

```python
from collections import Counter

def modified_precision(candidate, references, n):
    def ngrams(seq):
        return Counter(tuple(seq[i:i + n]) for i in range(len(seq) - n + 1))
    cand = ngrams(candidate)
    ceiling = Counter()
    for ref in references:
        for g, c in ngrams(ref).items():
            ceiling[g] = max(ceiling[g], c)
    clipped = sum(min(c, ceiling[g]) for g, c in cand.items())
    return clipped, sum(cand.values())

reference = "the dog bit the man".split()
candidate = "the man bit the dog".split()

for n in (1, 2, 3):
    hit, total = modified_precision(candidate, [reference], n)
    print(f"{n}-gram precision: {hit}/{total}")
```

Real output:

```
1-gram precision: 5/5
2-gram precision: 3/4
3-gram precision: 0/3
```

A sentence with the subject and object swapped scores 100% on unigrams and 75% on bigrams. Whether
that is a good score depends entirely on how the metric weights n-gram orders — which is a decision
made by whoever configured the evaluation, not a fact about the translation.

## Academic Sources

- ["Coping with Syntactic Ambiguity or How to Put the Block in the Box on the Table"](https://aclanthology.org/J82-3004.pdf) — Church & Patil (1982), Catalan-number ambiguity
- ["ELIZA — A Computer Program For the Study of Natural Language Communication Between Man and Machine"](https://dl.acm.org/doi/10.1145/365153.365168) — Weizenbaum (1966)
- ["Building a Large Annotated Corpus of English: The Penn Treebank"](https://aclanthology.org/J93-2004.pdf) — Marcus, Marcinkiewicz & Santorini (1993), corpus sizes and annotation rates
- ["BLEU: a Method for Automatic Evaluation of Machine Translation"](https://aclanthology.org/P02-1040.pdf) — Papineni et al. (2002)
- ["The Winograd Schema Challenge"](https://cdn.aaai.org/ocs/4492/4492-21843-1-PB.pdf) — Levesque, Davis & Morgenstern (2012), coreference that needs world knowledge
- ["Google's Neural Machine Translation System"](https://arxiv.org/abs/1609.08144) — Wu et al. (2016), the 60% error reduction and the BLEU-versus-human result
- ["GLUE: A Multi-Task Benchmark and Analysis Platform"](https://arxiv.org/abs/1804.07461) — Wang et al. (2018)
- ["SuperGLUE: A Stickier Benchmark"](https://arxiv.org/abs/1905.00537) — Wang et al. (2019), the saturation of GLUE
- ["Language Models are Few-Shot Learners"](https://arxiv.org/abs/2005.14165) — Brown et al. (2020), GPT-3
- ["The State and Fate of Linguistic Diversity and Inclusion in the NLP World"](https://arxiv.org/abs/2004.09095) — Joshi et al. (2020), the six language classes
- ["Understanding searches better than ever before"](https://blog.google/products/search/search-language-understanding-bert/) — Google (2019), BERT in Search

## Frequently Asked Questions

### Is NLP the same as LLMs?

No. NLP is a field — the set of problems involved in getting computers to work with human language, going back to the 1950s. A large language model is a technique, and currently the one that solves most of those problems best. Translation, summarisation and question answering are NLP tasks; an LLM is one way to do them, and until about 2018 each had its own separate model instead.

### What does an NLP engineer actually do now?

Mostly the work around the model rather than the model itself: choosing what to retrieve and put in the prompt, building evaluation sets that reflect the actual job, deciding which tasks justify a fine-tuned small model instead of an API call, and handling the languages and domains the frontier model is weak on. Training a task-specific parser or sentiment classifier from scratch is now the unusual case.

### Why is human language so hard for computers?

Because the ambiguity is structural, not occasional. Church and Patil showed in 1982 that a sentence with five prepositional phrases has 42 grammatically valid parses and one with nine has 4,862 — counts that follow the Catalan numbers. Readers never notice, because they resolve it with knowledge about the world that is nowhere in the sentence.

### Do NLP results in English transfer to other languages?

Not reliably. Joshi et al. surveyed 2,485 languages and found seven — English, Spanish, German, Japanese, French and two others — holding almost all the labelled and unlabelled resources, while 2,191 languages with a billion speakers between them had essentially none. Even within Google's own translation results the gap to human quality on English-to-Chinese was five times the gap on English-to-Spanish.

### Does a high benchmark score mean the model understands language?

It means it scores well on that benchmark. GLUE was published in April 2018 as a hard test of language understanding; by July 2019 the best system had passed the non-expert human baseline, which is why its authors immediately built a harder one. A score is evidence about a test, and tests saturate faster than capabilities improve.

## Related

### Related terms

- [Large Language Model (LLM)](https://howaiworks.ai/glossary/large-language-model)
- [Tokenization](https://howaiworks.ai/glossary/tokenization)
- [Transformer](https://howaiworks.ai/glossary/transformer)
- [Semantic Understanding](https://howaiworks.ai/glossary/semantic-understanding)
- [Text Analysis](https://howaiworks.ai/glossary/text-analysis)
- [Benchmark](https://howaiworks.ai/glossary/benchmark)

---

Source: https://howaiworks.ai/glossary/natural-language-processing — HowAIWorks.ai