Text Analysis

Definition

Text analysis is turning unstructured text — reviews, emails, support tickets, contracts, clinical notes — into structured information a program can filter, count and act on: a category label, a sentiment score, a list of the people and places mentioned, a set of topics. The input is prose meant for a human; the output is a row in a table. That conversion is the whole job, and it is why the field also goes by "text mining": you are mining a fixed quantity of structure out of text that was never written to contain it.

It helps to place text analysis against the broader field. Natural language processing is everything a computer might do with human language, generation included; text analysis is the reading half — the concrete tasks that take text in and emit structure, as opposed to text generation, which emits new language. So this page is deliberately a task catalog and a pipeline rather than a second history of NLP. The tasks below — classification, sentiment analysis, named-entity recognition, topic modeling, keyphrase extraction — were separate research problems for decades and are now often one prompt to a large language model, but they are still the units people name, budget and measure, and the thing that breaks is still specific to each one.

How It Works

Almost every text analysis task runs the same four-step pipeline, and naming the steps is the fastest way to see where a system goes wrong.

Raw text in. A single document — one email, one review, one paragraph of a note.

Tokenize. The string is split into units the model can index: words, or more commonly subword pieces. This is tokenization, and it is already lossy in ways that matter downstream — a tokenizer that splits covid-19 into covid, -, 19 has thrown away the fact that it was one thing.

Represent. The tokens become numbers, because a model does arithmetic, not reading. There are two families, and the choice decides what the system can possibly notice. The older bag-of-words family (including TF-IDF) counts which words appear and discards the order entirely — under it, the dog bit the man and the man bit the dog produce the identical vector, which is fine for "is this email about shipping?" and fatal for anything where word order is the meaning. The modern family is embeddings: each token, or the whole document, becomes a dense vector positioned so that texts with similar meaning sit close together, and word order is preserved through the model that produces them.

Task-specific model, then structured output. A classifier maps the representation to a label; a sequence tagger maps each token to a tag; a topic model factorises a whole collection at once. What comes out is the structure: spam, or 0.91 positive, or the span London → LOCATION.

Named-entity recognition is the clearest place to see the structured output, because it is per-token. The standard scheme tags each token B- (beginning of an entity), I- (inside one) or O (outside any), so a six-token sentence becomes six tags and three entity spans fall out of them:

token:  Apple    hired   Jane      Smith    in   London
tag:    B-ORG    O       B-PER     I-PER    O    B-LOC
                         └─── one PERSON span ───┘

Three entities — one organisation, one two-token person, one location — extracted from a sentence that arrived as an undifferentiated string. That table is the entire point of text analysis: a program can now sort by organisation, and it could not before.

Types

The task names below are a real typology — these are the words practitioners, datasets and job adverts actually use — and they differ in exactly one place: what counts as the structured output.

Text classification

The workhorse. One document in, one label out, from a fixed set: spam/not spam, billing/technical/sales, English/French. Because the output is a single discrete choice, classification is scored with precision, recall and F1 (worked below) and is the task most often left to a small fine-tuned model even in the LLM era, because you can run it on millions of items for the cost of one API call each. See classification for the general mechanism.

Sentiment analysis

A special case of classification where the label is polarity — positive, negative, neutral, or a score — usually over opinions: reviews, tweets, survey responses. It looks easy and is not, because the signal lives in negation, comparison and tone rather than in the presence of "good" or "bad" words, which is where most of the failures in the Challenges section come from.

Named-entity recognition (NER)

Instead of one label for the document, a label for each span: which tokens name a person, an organisation, a location, a date, a monetary amount. The BIO tagging shown above is the mechanism. NER is what converts a sentence into rows in a database, so it sits underneath knowledge graphs, compliance monitoring and document search.

Topic modeling

The one unsupervised member of the catalog. Given a large collection and no labels, discover the recurring themes and say which documents belong to which — classic algorithms like Latent Dirichlet Allocation treat each document as a mixture of topics and each topic as a distribution over words. Because nobody told it what the topics are, the output is a set of word clusters a human still has to name, which is both its strength (it finds themes you did not know to look for) and its weakness (some clusters are noise).

Keyword and keyphrase extraction

Pull the handful of terms that most characterise a document — for indexing, tagging or a summary line. The output is a ranked list of phrases lifted from the text, which distinguishes it from text generation: extraction never writes a word the document did not already contain.

Real-World Applications

Spam and abuse filtering is text classification at planetary scale. Gmail routes mail into inbox versus spam for billions of accounts, and content-moderation systems triage posts before a human ever sees them. This is precisely the setting where a small, fast classifier beats a frontier model: the per-item cost has to be a fraction of a cent, and the label is all anyone needs.

Support-ticket routing turns free-text complaints into a queue: classify the ticket by department and urgency, extract the product name and order number with NER, and the ticket lands on the right desk pre-tagged. The structured output is the routing decision.

Legal and financial document review. E-discovery and contract review use NER and classification to find every clause of a type, every named counterparty, every date and dollar amount across millions of pages — the volume that makes reading by hand impossible is exactly the volume text analysis exists for.

Clinical documentation. Extracting diagnoses, medications and dosages from free-text notes into a structured record is a large, real deployment — see ambient clinical documentation for the version that listens to the visit itself — and, as the Challenges note, the domain where a sentiment model trained on product reviews reads the polarity backwards.

Voice-of-customer analytics. Aggregating sentiment and topics across thousands of reviews, survey answers and social posts to answer "what are people unhappy about this week?" — topic modeling to find the themes, sentiment analysis to score them, no single document mattering on its own.

Challenges

Negation and sarcasm invert the label without changing the vocabulary. "Not bad" is positive and contains the strongest negative word in the sentence; "yeah, great, another outage" is negative and contains "great". Any representation that weakens word order — bag-of-words most of all — literally cannot see the "not" that flips "bad", and tone is not in the words at all. This is why sentiment analysis is harder than its accuracy numbers on clean review data suggest.

Domain shift breaks a model quietly. A sentiment classifier trained on product reviews carries the priors of that domain into every new one. Point it at radiology reports and it fails on the first line, because in clinical language "negative" and "unremarkable" are good news — no disease found — the exact opposite of their polarity in a review. The model is not broken; it is answering a different domain's question, and nothing in the output announces the mismatch.

Accuracy lies when the classes are imbalanced. If 80 of 100 emails are legitimate, a model that blindly calls everything "not spam" scores 80% accuracy while catching zero spam. That is why the worked example below reports precision, recall and F1 instead: a 90%-accurate filter can still be quarantining 6 real emails and missing 4 real spam, and only the per-class numbers show it.

Label ambiguity caps the achievable score. Human annotators disagree about sentiment on the same sentence surprisingly often, and a model cannot be more consistent than the labels it learned from. When two reasonable people tag the same review differently, "the right answer" is not a single value, and a benchmark that pretends it is will punish a correct model for a defensible call.

Representation is a lossy commitment. Tokenization decisions upstream — splitting covid-19, merging or dropping punctuation, casing — are made before any task model sees the text, and they silently cap what the task can recover. A pipeline is only ever as good as its earliest step, and that step is the one nobody looks at when the accuracy is disappointing.

Code Example

The most useful arithmetic in text analysis is the classification scorecard, because it is where "my model is 90% accurate" turns out to mean something much narrower. Take a spam filter run over 100 emails, of which 20 are truly spam. It correctly flags 16 of them (true positives), misses 4 (false negatives), wrongly flags 6 real emails (false positives) and leaves 74 alone (true negatives). Every metric below is derivable from those four counts alone — no data, no model, just the confusion matrix:

# A spam filter's scorecard on 100 emails: 20 are really spam, 80 are real mail.
TP, FP, FN, TN = 16, 6, 4, 74      # true/false positives, false/true negatives
n_spam = TP + FN                    # 20 actual spam
n_ham  = FP + TN                    # 80 actual ham

precision = TP / (TP + FP)          # of the ones flagged, how many were spam
recall    = TP / (TP + FN)          # of the real spam, how many we caught
f1        = 2 * precision * recall / (precision + recall)
accuracy  = (TP + TN) / 100
baseline  = n_ham / 100             # "call everything ham" and never flag

print(f"precision = {TP}/{TP+FP} = {precision:.3f}")
print(f"recall    = {TP}/{n_spam} = {recall:.3f}")
print(f"F1        = {f1:.3f}")
print(f"accuracy  = {accuracy:.2f}")
print(f"always-ham baseline accuracy = {n_ham}/100 = {baseline:.2f}")

Running it prints:

precision = 16/22 = 0.727
recall    = 16/20 = 0.800
F1        = 0.762
accuracy  = 0.90
always-ham baseline accuracy = 80/100 = 0.80

Read those five numbers together and the headline dissolves. The filter is "90% accurate", but a model that does nothing at all — flags no email ever — is already 80% accurate here purely because most mail is legitimate, so the real lift is 10 points, not 90. Of the 22 emails it did flag, only 16 were spam, so precision is 0.727: 6 real emails went to the spam folder, the error a user actually notices. It caught 16 of 20 spam, so recall is 0.800: 4 spam reached the inbox. F1, the harmonic mean, lands at 0.762 and refuses to let either number hide behind the other — which is the whole reason text analysis reports it. Change the threshold to flag more aggressively and recall rises while precision falls; the single accuracy figure would move barely at all and tell you none of this.