Natural Language Processing (NLP)

NLP is the field; large language models are now the answer to most of it. What sixty years of work was for, and which parts are still unsolved.

Published Updated

On this page

Definition

Natural language processing is the field of getting computers to do useful things with human language — and since roughly 2020, one kind of model has been the answer to most of it. That is the first thing to settle, because it is the live confusion: NLP is a set of problems, a large language model is the current answer to them. Asking whether NLP and LLMs are the same thing is like asking whether medicine and antibiotics are the same thing. Translation, summarisation, entity extraction and question answering are NLP tasks and were NLP tasks decades before a transformer existed; an LLM is simply the technique that now does them.

Here is what those problems actually look like, on numbers. Take the sentence Kenneth Church and Ramesh Patil put in the title of their 1982 paper: Put the block in the box on the table. It has two readings — put the block into the box that is on the table, or put the block that is in the box onto the table. Nothing in the words decides it; you decide by knowing things about blocks, boxes and tables. Add a third prepositional phrase and there are 5 readings, a fourth gives 14, a fifth gives 42. The counts are the Catalan numbers, and the paper reports that their parser found exactly that many parses for real sentences all the way out to nine prepositional phrases — 4,862 structurally valid analyses of a sentence a person reads once without noticing any ambiguity at all.

That gap — between what a sentence formally permits and what a reader effortlessly picks — is the whole of NLP in one line. Sixty years of the field is a sequence of attempts to close it. Hand written rules closed it for a paragraph at a time and then broke. Annotated corpora and one statistical model per task closed it further, at enormous human cost. What finally moved it was scale: GPT-3 (Brown et al., 2020) noted that the prevailing method "still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples", then showed a single 175-billion-parameter model doing many of those same tasks from a handful of examples typed into the prompt, with no gradient updates at all.

The tasks did not disappear. The separate models mostly did — which is exactly why a 2026 job advert for an NLP engineer still lists named-entity recognition, part-of-speech tagging, parsing and sentiment analysis as if they were separate skills. They were, for thirty years. Knowing which of them still needs its own model is now a large part of the job.

And here is what breaks if you get the distinction wrong: you start reading benchmark scores as capability claims. A model that tops an English leaderboard has demonstrated something about that leaderboard, in that language. It has not demonstrated that it will parse your clinical notes, or that it works in Yoruba, or that the metric you scored it with measures what you care about. Every one of those has a measured counter-example further down this page.

How It Works

Three problems that turned out to be hard for non-obvious reasons

Ambiguity is combinatorial, not occasional. The Catalan explosion above is the structural case, and lexical ambiguity multiplies on top of it: every word with several senses forks the reading again. The counter-intuitive part is not that ambiguity exists but that people never see it. A system that enumerates possible analyses does the linguistically correct thing and drowns; a reader who does the linguistically incorrect thing — commits instantly to one reading using world knowledge — is right almost always.

Reference has no local answer. Consider the pair Levesque, Davis and Morgenstern built their 2012 challenge around, adapted from Terry Winograd's 1972 work: The town councillors refused to give the angry demonstrators a permit because they feared violence versus the same sentence with advocated. Change one word and the pronoun they flips from the councillors to the demonstrators. No amount of grammatical analysis of the sentence helps, and no co-occurrence statistic over a corpus helps either, because the two sentences are nearly identical strings. You need a model of who fears what.

Context reaches outside the sentence entirely. Google's own illustration when it deployed BERT in Search in October 2019 was the query 2019 brazil traveler to usa need a visa. The word "to" carries the whole meaning — who is travelling where — and Google's blog post is blunt that its earlier algorithms "wouldn't understand the importance of this connection, and we returned results about U.S. citizens traveling to Brazil". The information that resolves the query is a function word that a keyword system throws away.

The task list, and what happened to it

The vocabulary a newcomer meets — tokenisation, part-of-speech tagging, syntactic parsing, named-entity recognition, coreference resolution, sentiment classification, machine translation, summarisation, question answering — is a list of tasks, not a taxonomy of the field. Each was once a research community with its own annotated dataset, its own model architecture and its own leaderboard. A 2015 production pipeline genuinely ran several of them in sequence: tokenise, tag, parse, then extract.

Most of that pipeline is now one prompt. What survives as separate machinery is the parts that are either cheaper or more controllable done separately: tokenization still happens, because it is how text becomes numbers at all; retrieval and ranking still run on small embedding models because you cannot afford to call a frontier model once per document in an index; and high-volume classification — spam, moderation, routing — often stays with a small fine-tuned encoder for the same reason.

Three eras, and what each one cost

Rules, written by hand. Joseph Weizenbaum's ELIZA (1966) is the honest specimen, because its author documented exactly how little was inside it. Input was matched against keywords, which triggered decomposition rules, which fired reassembly rules to build a reply. Weizenbaum's own assessment is the part worth keeping: the program "maintains the illusion of understanding with so little machinery", and "throws away each of its inputs" except the fragments it transforms. His test for the real thing is still a good one — "the crucial test of understanding... is not the subject's ability to continue a conversation, but to draw valid conclusions from what he is being told."

Annotate a corpus, train one model per task. The Penn Treebank is the monument of this era. Marcus, Marcinkiewicz and Santorini report that its first phase (1989–1992) produced 4,885,798 tokens tagged for part of speech and 2,881,188 tokens annotated with syntactic structure. The cost is the interesting number. Annotators bracketing sentences entirely by hand managed about 375 words per hour after three weeks; correcting a parser's output instead, they reached 750 to over 1,000 words per hour after three or four months' experience. At 750 words an hour, the parsed portion alone represents roughly 3,800 annotator-hours — about two person-years of skilled linguistic labour for under three million words.

Pretrain on unlabelled text, then prompt. Llama 3's flagship was trained on 15.6 trillion tokens. At the 3.94 characters per token its own paper measures, that is about 61 trillion characters, or on the order of 10 trillion words — roughly 3.5 million times the size of the Penn Treebank's parsed portion. Hand-annotating a corpus that size at the treebank's best sustained rate would take about seven million person-years. That arithmetic is the entire argument for self-supervised learning: the supervision has to come from the text predicting itself, because there is no version of the world in which it comes from people.

What the model is doing, in one paragraph

Text is split into subword units by a tokenizer, each unit becomes a vector, and a stack of transformer layers uses attention to let every position draw on every earlier one — which is the mechanism that finally handled reference and long-range context without anyone writing a rule for them. The model then predicts one token at a time. The large language model page covers that loop, its costs and where its knowledge lives; the point for NLP is narrower and stranger. A single objective — predict the next token — turned out to subsume a field's worth of separately-defined tasks, and nobody designed it that way.

Real-World Applications

Search. Google announced BERT in Search on 25 October 2019 with the claim that it "will help Search better understand one in 10 searches in the U.S. in English". That is a deployment number, not a benchmark: a transformer language model placed between a billion daily queries and their results. Semantic search and retrieval-augmented generation are the same idea grown up — the query is understood by a model, and the documents are found by comparing vectors rather than matching words.

Machine translation. Google's 2016 GNMT paper is the cleanest before-and-after in the field's history: replacing the phrase-based production system with a neural one "reduces translation errors by an average of 60%" on a human side-by-side evaluation. Their production table puts GNMT at 5.428 against a human translator's 5.504 on English-to-Spanish, and 4.594 against 4.987 on English-to-Chinese, on a six-point scale. DeepL built a business on the same architecture. Note what the two pairs of numbers already show: the residual gap to human quality on English-to-Chinese is more than five times the gap on English-to-Spanish, in the same system on the same day.

General assistants. ChatGPT, Claude and Gemini are the visible face of the collapse described above: one model, prompted differently, doing summarisation, translation, extraction and classification without a task-specific model behind any of them. This is also why "NLP product" stopped being a useful category — a legal-summary tool and a customer-support router are now the same model with different context.

Speech and clinical documentation. Voice recognition and text-to-speech converged on the same architecture from the audio side, which is what makes ambient clinical documentation possible: a microphone in the consulting room, a transcript, and a structured note generated from it. The NLP problem there is not transcription but knowing which parts of a conversation belong in a medical record.

The invisible majority. Spam filtering, content moderation, support-ticket routing, resume screening and product-review analysis run at volumes where calling a frontier model per item is absurd. These mostly still use small fine-tuned classifiers, and they are the largest deployment of NLP by item count. A model that reads a document once and emits a label does a fraction of the work of one that reads a prompt template plus the document and then writes a sentence.

Challenges

Automatic metrics reward surface overlap, and everyone knows it. BLEU (Papineni et al., 2002) scores a translation by counting n-gram matches against references. Run it on the dog bit the man scored against the reference the dog bit the man reversed — that is, the candidate the man bit the dog — and modified unigram precision is 5/5, a perfect score for a sentence that says the opposite thing. Bigram precision is 3/4 and it only collapses at the trigram level. The BLEU authors were candid about the failure mode in the original paper: a system producing the fluent "East Asian economy" is "penalized heavily on the longer n-gram precisions if all the references happen to read 'economy of East Asia'."

Optimising the metric is not the same as improving the system, and there is a measurement of it. The GNMT team refined their models with reinforcement learning against BLEU directly. BLEU rose from 40.35 to 41.16 — +0.81 points — while the human side-by-side score went from 4.46 to 4.44, fractionally down. Their own conclusion is that "even though RL refinement can achieve better BLEU scores, it barely improves the human impression of the translation quality", and they stopped using it in production experiments. When you optimise a proxy hard enough, you get the proxy.

Benchmarks saturate faster than capabilities improve. GLUE, nine language-understanding tasks, was published in April 2018. By early July 2019 the state of the art stood at 88.4 against a non-expert human baseline of 87.1, exceeding it on four of the nine tasks — so its authors shipped SuperGLUE, on which the human estimate was 89.8 and a strong BERT baseline managed 71.5. The lesson is not that the benchmark was bad. It is that a benchmark measures a test, that the gap between test and capability is invisible from the score, and that fifteen months is roughly how long a hard NLP benchmark now lasts.

English results do not transfer, and the resource gap is not close to uniform. Joshi et al. classified 2,485 languages by how much labelled and unlabelled data exists for each. Seven languages — English, Spanish, German, Japanese, French among them — make up 0.28% of the total and take the benefit of every advance. 2,191 languages, 88.17% of the total and about a billion speakers, sit in the bottom class with effectively no representation in any resource they measured. A frontier model's multilingual claims are downstream of that distribution, not independent of it.

The model cannot see characters. Counting letters, detecting rhyme, spotting a typo and reasoning about spelling all fail for a reason that has nothing to do with language understanding: the units the model receives are subword ids, not letters. This is a tokenization property rather than an NLP one, but it produces some of the most confidently-cited "proof that models don't understand language", which is why it is worth knowing where the failure actually lives.

Fluent output is not calibrated output. Nothing in next-token prediction produces a confidence signal a reader can use, so a wrong extraction reads exactly like a right one. For classification at volume this is the strongest remaining argument for a small supervised model over a prompt: a fine-tuned classifier gives you a probability you can threshold, and a generated sentence gives you prose. Hallucination is the same property showing up in longer form.

Future Trends

Evaluation is the field's live research problem. With benchmarks saturating in about a year, the work has moved to adversarially-constructed test sets, held-out private evaluations, and model-graded rubrics — each of which has its own failure mode, and none of which has the clean reproducibility that made BLEU and GLUE so useful. Expect the credible claims to come with a described evaluation set rather than a single number.

Low-resource languages are the largest unbanked value in the field. The route is no longer "collect a parallel corpus for each pair" but transfer: pretrain multilingually, then rely on representations shared across languages to carry a task learned in one into another. It works unevenly, and the two levers that matter most are unglamorous — the tokenizer's allocation of vocabulary across writing systems, and the amount of unlabelled text in the language at all. The 2,191 languages in Joshi's bottom class do not have Wikipedia editions to pretrain on.

Meaning grounded in something other than text. The tasks NLP has not solved cluster around reference to things outside the document: pronouns across a hundred-page contract, a claim that must be checked against a database, an instruction whose correctness depends on the state of the world. Multimodal training and tool use are both attempts to attach language to a referent rather than to more language. This is the oldest open problem in the field, and it is the one a larger pretraining run has so far moved least.

Small models coming back for narrow work. Distillation plus fine-tuning keeps producing task models that match a frontier model on one task at a fraction of the cost per item, which matters precisely where volume is high and the task is stable. The pendulum swung all the way to "one general model for everything" and is settling somewhere sensible: general models for the long tail, specialised ones for the head.

Code Example

The ambiguity explosion in the Definition is worth reproducing, because it is a fact about English structure rather than about any model. The number of ways to bracket n prepositional phrases is the number of binary trees over n + 1 leaves:

from functools import lru_cache

@lru_cache(None)
def bracketings(leaves):
    """Ways to fully parenthesize a sequence of `leaves` items."""
    if leaves == 1:
        return 1
    return sum(bracketings(i) * bracketings(leaves - i) for i in range(1, leaves))

for pps in range(1, 10):
    print(f"{pps} PPs -> {bracketings(pps + 1):>5} parses")

Real output:

1 PPs ->     1 parses
2 PPs ->     2 parses
3 PPs ->     5 parses
4 PPs ->    14 parses
5 PPs ->    42 parses
6 PPs ->   132 parses
7 PPs ->   429 parses
8 PPs ->  1430 parses
9 PPs ->  4862 parses

Those are the Catalan numbers, and the 4,862 at nine phrases is the figure Church and Patil report their parser actually returning. The second block computes BLEU's modified n-gram precision on the word-order example from the Challenges section:

from collections import Counter

def modified_precision(candidate, references, n):
    def ngrams(seq):
        return Counter(tuple(seq[i:i + n]) for i in range(len(seq) - n + 1))
    cand = ngrams(candidate)
    ceiling = Counter()
    for ref in references:
        for g, c in ngrams(ref).items():
            ceiling[g] = max(ceiling[g], c)
    clipped = sum(min(c, ceiling[g]) for g, c in cand.items())
    return clipped, sum(cand.values())

reference = "the dog bit the man".split()
candidate = "the man bit the dog".split()

for n in (1, 2, 3):
    hit, total = modified_precision(candidate, [reference], n)
    print(f"{n}-gram precision: {hit}/{total}")

Real output:

1-gram precision: 5/5
2-gram precision: 3/4
3-gram precision: 0/3

A sentence with the subject and object swapped scores 100% on unigrams and 75% on bigrams. Whether that is a good score depends entirely on how the metric weights n-gram orders — which is a decision made by whoever configured the evaluation, not a fact about the translation.

Academic Sources

Frequently Asked Questions

No. NLP is a field — the set of problems involved in getting computers to work with human language, going back to the 1950s. A large language model is a technique, and currently the one that solves most of those problems best. Translation, summarisation and question answering are NLP tasks; an LLM is one way to do them, and until about 2018 each had its own separate model instead.
Mostly the work around the model rather than the model itself: choosing what to retrieve and put in the prompt, building evaluation sets that reflect the actual job, deciding which tasks justify a fine-tuned small model instead of an API call, and handling the languages and domains the frontier model is weak on. Training a task-specific parser or sentiment classifier from scratch is now the unusual case.
Because the ambiguity is structural, not occasional. Church and Patil showed in 1982 that a sentence with five prepositional phrases has 42 grammatically valid parses and one with nine has 4,862 — counts that follow the Catalan numbers. Readers never notice, because they resolve it with knowledge about the world that is nowhere in the sentence.
Not reliably. Joshi et al. surveyed 2,485 languages and found seven — English, Spanish, German, Japanese, French and two others — holding almost all the labelled and unlabelled resources, while 2,191 languages with a billion speakers between them had essentially none. Even within Google's own translation results the gap to human quality on English-to-Chinese was five times the gap on English-to-Spanish.
It means it scores well on that benchmark. GLUE was published in April 2018 as a hard test of language understanding; by July 2019 the best system had passed the non-expert human baseline, which is why its authors immediately built a harder one. A score is evidence about a test, and tests saturate faster than capabilities improve.

Related Terms

Large Language Model (LLM)

A large language model predicts the next token of text. How that produces answers, where its knowledge is stored, and why an LLM is not a chatbot.

Tokenization

How text becomes the integers a language model reads: BPE, WordPiece and Unigram, why " strawberry" is one token, and what that breaks.

Transformer

The 2017 neural-network architecture that processes a whole sequence in parallel using self-attention instead of recurrence — the basis of modern LLMs.

Semantic Understanding

Whether an AI grasps meaning well enough to act correctly on novel inputs, or is pattern-matching that breaks on rephrasing — and why that line is contested.

Text Analysis

Turning unstructured text into structured data — the concrete NLP task catalog: classification, sentiment analysis, entity extraction and topic modeling.

Benchmark

An AI benchmark is a standardized test or dataset used to evaluate and compare the performance of different AI models across tasks like reasoning and coding.

Continue Learning

Explore our use-case guides and prompts to deepen your AI knowledge.

Browse AI Tools← Back to Glossary