Definition
Data analysis is the work of getting from a table of records to a number somebody will act on: deciding what question the data can actually answer, looking at the distribution before computing anything, choosing a summary that does not misrepresent it, saying how uncertain the result is, and stating what should change as a consequence. Most of the difficulty is not statistical machinery — it is that the same honest arithmetic on the same rows produces different answers depending on the grain you aggregate at and the denominator you divide by, and nothing in the data tells you which one your decision needs.
It is distinguished from its two nearest neighbours by what it optimises. A machine learning model is judged by how well it predicts records it has never seen, and is allowed to be an uninterpretable black box as long as the held-out score holds. An analysis is judged by whether a human makes a better decision after reading it, which makes interpretability and honest uncertainty part of the deliverable rather than a nice-to-have. Data science is the umbrella term that covers both, plus the engineering of shipping the model. And the data processing that gets the table into a queryable state — the pipelines, the cleaning, the schemas — is upstream of all of it: this page starts at the point where the table exists and someone has a question.
How It Works
The question comes before the query
"Explore the data and see what you find" is how analysis is described and not how it works. A usable analytic question names three things: the population (which rows count), the metric (what is measured, at what grain), and the decision (what would be done differently under each possible answer). "Is churn up?" fails all three. "Among accounts that signed up at least 90 days ago, did the share cancelling within their first 90 days rise between the Q1 and Q2 cohorts by enough to justify pausing the pricing change?" answers all three, and it does so before any SQL is written, which is the point — the third clause is what stops you producing a beautifully correct number nobody acts on.
Mechanically the work is mostly aggregation and comparison: SQL against a warehouse for the grouping, pandas or dplyr or DuckDB for reshaping, a plotting library to look at what came back, a notebook holding the sequence so someone can re-run it. That tooling has been stable for years and is not the hard part. The hard part is in the next four subsections.
Description, inference and prediction are three different claims
These get conflated constantly, and the conflation is where confidently wrong conclusions start.
A descriptive claim is about the rows you have: "revenue in June was $4.1M." It has no uncertainty attached because nothing was sampled — you counted. An inferential claim is about a population you only partly observed: "our conversion rate is about 2.4%." This is an estimate from a sample and is meaningless without an interval. A predictive claim is about rows that do not exist yet: "July revenue will be $4.3M." It needs both an interval and an assumption that the future resembles the past.
The common failure is reading a descriptive number as an inferential one. A dashboard tile saying "conversion: 2.4%, up from 2.0%" is describing what happened to a particular set of visitors last week. Treating it as a statement about how the product converts requires the arithmetic in the next-but-one subsection, and that arithmetic usually says the movement is noise.
Look before you aggregate
Frank Anscombe published four small datasets in The American Statistician in 1973 that share nearly every summary statistic anyone would report: mean of x = 9.0, variance of x = 11.0, mean of y = 7.50, variance of y = 4.12, correlation 0.816, and the identical fitted line y = 3.00 + 0.500x. Plotted, they are a rough linear relationship, a clean parabola, a perfect line with one outlier dragging it, and a vertical stack of points at one x-value with a single distant point setting the entire slope. Four completely different situations, each calling for a different decision, all reported identically.
Matejka and Fitzmaurice extended this at CHI in 2017 with the Datasaurus Dozen: thirteen datasets agreeing to two decimal places on means, standard deviations and correlation, one of which is a picture of a dinosaur. The lesson is durable and specific — a summary statistic is a lossy compression, and you cannot tell from the summary how much it lost. Exploratory data analysis, the practice John Tukey named in his 1977 book, is the habit of examining distributions, ranges, missing values and pairwise plots before choosing what to report, precisely because the choice cannot be made responsibly from the summaries themselves.
Choosing a summary that does not lie
Take ten orders from one day, in dollars: 20, 22, 25, 28, 30, 32, 35, 40, 45, and one enterprise order of 1,723. The total is $2,000, so the mean order value is $200. The median is the midpoint of 30 and 32, or $31. The mean is 6.5 times the median, and it is larger than nine of the ten actual orders.
Neither is wrong; both are correct arithmetic on the same ten rows, and the divergence is itself diagnostic. It is what right skew looks like, and business metrics — order value, session duration, customer lifetime value, API latency, ticket resolution time — are nearly all right-skewed, because they are bounded below at zero and unbounded above.
Which one a decision should use follows from one question: what does the decision multiply by? If the outcome scales with the total, use the mean, because total = mean × count and the median carries no information about the total. Forecasting revenue, sizing inventory, provisioning capacity and budgeting cost all scale with the total, so all take the mean. If the decision concerns the experience of a typical unit, use the median or a stated percentile. Setting free shipping at "just above average order value" would put the threshold at $200 and leave 90% of that day's customers unaffected; setting it just above the median puts it near $40, where it changes what most people do.
Latency makes the same point at the other tail. Google's Site Reliability Engineering practice defines latency objectives on percentiles rather than means for a reason Jeffrey Dean and Luiz Barroso set out in The Tail at Scale (CACM, 2013): if one user request fans out to 100 backend servers in parallel and waits for all of them, then 1 − 0.99¹⁰⁰ = 63% of requests will be slower than the 99th-percentile latency of a single server. A mean latency that looks healthy describes an experience most users are not having. (Why fan-out compounds tails this way, and what engineers do about it, belongs to distributed computing; the point here is only that the summary you report decides whether you can see it.)
What an interval actually spans
Suppose 1,000 visitors and 24 conversions — a measured rate of 2.4%. The standard error of a proportion is √(p(1−p)/n) = √(0.024 × 0.976 / 1000) = 0.0048, or 0.48 percentage points. The 95% interval is 2.4% ± 1.96 × 0.48%, which is roughly 1.5% to 3.4%. The top of that range is more than twice the bottom.
Now put last week's 2.0% beside it. Its interval on the same traffic runs from about 1.1% to 2.9%. The two intervals overlap across nearly their whole length, so "conversion improved from 2.0% to 2.4%" is a sentence about sampling noise dressed as a result. Interval width scales as 1/√n, so cutting the uncertainty in half means quadrupling the sample: 4,000 visitors narrows the interval to about ±0.47 points, still wide enough that a 0.4-point move is not resolvable.
Two things this arithmetic does not give you, and which are deliberately not on this page: how many users you need to reliably detect a difference of a given size, and whether a difference you did detect was caused by the change you shipped. Both belong to causal reasoning — the second one especially, because an interval that excludes zero says the difference is unlikely to be noise, and says nothing whatever about what produced it.
Granularity and denominators change the answer
Region A has 100 visitors and 5 conversions, a 5% rate. Region B has 10,000 visitors and 100 conversions, a 1% rate. Average the two rates and you get 3.0%. Pool the raw counts and you get 105 / 10,100 = 1.04%. The first number is nearly three times the second, and both came out of a GROUP BY.
The average-of-averages is the wrong one here because it weights a region of 100 people equally with a region of 10,000. This is not Simpson's paradox — no confounder is involved and nothing is hidden in a subgroup; it is division performed in the wrong order, which is why it survives code review so easily. A mean of ratios is a different quantity from a ratio of sums, and only one of them answers "what fraction of our visitors converted."
The same choice recurs as grain. A metric defined per session is a different metric from one defined per user, and a company with heavy repeat usage can watch per-session engagement fall while per-user engagement rises. Aggregating a weekly total hides a weekday/weekend split that may be the entire story. And the denominator is where the largest errors live: reported COVID-19 case fatality ratios differed by an order of magnitude across countries in the first months of 2020 — above 10% in Italy, under 1% in Germany — and most of that gap was testing policy determining who entered the denominator, not the disease behaving differently.
Types
The four-stage ladder — descriptive (what happened), diagnostic (why it happened), predictive (what will happen) and prescriptive (what to do about it) — is a genuine industry taxonomy, popularised by Gartner and now the standard vocabulary of BI vendors, job descriptions and analytics roadmaps. It is worth knowing because you will be asked which stage your team is at.
It is also routinely oversold as a maturity ladder, and it is worth being precise about why. Only the first stage is achievable from data alone. Diagnostic analysis — attributing a metric movement to a cause — requires an assumption the data cannot supply, which is the entire subject of causal reasoning; dashboards labelled "root cause analysis" mostly perform correlation and let the reader do the attribution silently. Predictive analysis is not a harder version of diagnostic analysis but a different question: models built with regression or time series methods can forecast accurately while encoding no causal structure at all, which is why they are safe to anticipate with and unsafe to intervene with.
"Prescriptive" in most vendor material means an optimiser with a hand-written objective attached to a predictive model — real and useful, since pricing, routing and scheduling systems run this way, but only as good as the objective a person wrote and the causal assumption underneath the prediction. Treat the four stages as four kinds of question with different evidence requirements, not four rungs a team climbs.
Real-World Applications
Oakland Athletics, 2002. Billy Beane and Paul DePodesta replaced batting average with on-base percentage as the metric the front office bought players on, because on-base percentage tracked runs scored far more closely and was systematically underpriced by the rest of the league. The analysis was a change of summary statistic, nothing more exotic, and the team won 103 games on roughly a third of the New York Yankees' payroll. It is the cleanest example of the point in the mean-versus-median section: the metric you choose determines the decision, before any modelling begins.
Reinhart and Rogoff, 2010, and its correction. "Growth in a Time of Debt" reported that countries with public debt above 90% of GDP averaged −0.1% real growth, a finding cited heavily in support of austerity programmes across Europe and the United States. In 2013 Herndon, Ash and Pollin obtained the spreadsheet and found three problems: an Excel range that omitted five countries, selective exclusion of available country-years, and a weighting scheme in which one country's single bad year counted as much as another's nineteen-year average. Corrected, the figure became +2.2%. Every one of those three is an analysis error of the kind this page describes — grain, inclusion criteria, weighting — and none required a statistical mistake.
John Snow's cholera map, London 1854. Parish-level death totals showed nothing usable. Snow plotted deaths at the grain of individual street addresses, and a cluster around the Broad Street pump became visible that no district-level aggregate could have shown. It is the founding demonstration that granularity is not a presentation detail; the causal argument he built on top of it belongs to causal reasoning, but the map itself is exploratory data analysis.
Latency SLOs in production engineering. Every major service operator now states and alerts on latency objectives as percentiles — p50, p95, p99 — rather than averages. That convention exists because of the fan-out arithmetic above: an analytic insight about summary statistics hardened into standard operating practice.
Challenges
Testing enough segments guarantees a finding. Slice a flat result by 20 segments — country, device, plan tier, acquisition channel — and test each at the 5% level. If nothing is going on anywhere, the probability of at least one "significant" result is 1 − 0.95²⁰ = 64%. Slice 40 ways and it is 87%. The analyst does nothing dishonest; they look until something appears, and something always appears. This is why a segment discovered during exploration has to be re-tested on fresh data before it is acted on, and why a pre-registered question is worth more than a discovered one.
Choosing which rows to plot can kill people. On the night before the Challenger launch in January 1986, engineers arguing against flying in the cold presented O-ring damage data from the seven previous flights that had shown thermal distress. The 17 flights with no damage were left off the chart. In the seven-flight subset the relationship with temperature is invisible; across all 24 flights it is stark, and the launch temperature of 36°F was 17 degrees below the coldest previous launch. Edward Tufte's reconstruction in Visual Explanations (1997) is the standard treatment. The general failure is conditioning on the outcome you are studying — plotting only the failures, surveying only current customers, analysing only accounts that survived to today — and it silently deletes the comparison that would have carried the signal.
A correct number that arrives after the decision is worth nothing. The most common way an analysis fails in practice is not statistical at all: it answers a question that was interesting rather than one that was pending, or arrives a week after the call was made. This is what the third clause of a well-posed question — what would be done differently — is defending against. If no answer changes an action, the analysis is a report, and it should be scheduled as one rather than staffed as an investigation.
Future Trends
Natural-language querying moves the failure mode, it does not remove it. A large language model writing SQL against a warehouse is good at syntax and weak at semantics: it will produce a query that runs cleanly, returns a plausible number, and answers a subtly different question — the wrong grain, an inner join that silently drops rows without a match, a denominator that includes internal traffic. The old failure mode was a query that errored out; the new one is a confident number with no traceback, which is strictly harder to catch. The benchmark evidence is consistent with this: on BIRD-SQL, a text-to-SQL benchmark built on realistic dirty databases, the human execution-accuracy baseline is 92.96%, and the gap between that and automated systems is dominated by exactly these semantic errors rather than by malformed SQL.
Semantic layers are the industry's answer. Defining each metric once — its grain, its filters, its denominator — in a versioned layer that both dashboards and LLM query tools must go through (dbt's semantic layer, Cube, LookML) is a direct response to the average-of-averages and grain problems above. It does not make the choices for you; it makes them explicit and reviewable in one place instead of re-litigated in every query.
The scarce skill shifts from producing numbers to trusting them. As generating a plausible chart approaches zero cost, the bottleneck moves to judgement: whether the grain matches the decision, and whether the interval permits the conclusion. Automation has made that part of the job more valuable, not less.
Code Example
The average-of-averages error, side by side with the correct pooled rate:
import pandas as pd
df = pd.DataFrame({
"region": ["A", "B"],
"visitors": [100, 10_000],
"conversions": [5, 100],
})
df["rate"] = df.conversions / df.visitors # 0.05 and 0.01
# Wrong: a mean of ratios weights a 100-visitor region like a 10,000-visitor one
print(df.rate.mean()) # 0.03 -> "3.0%"
# Right: a ratio of sums answers "what share of visitors converted?"
print(df.conversions.sum() / df.visitors.sum()) # 0.0104 -> "1.04%"
# Weighting the mean by the denominator recovers the same answer
print((df.rate * df.visitors).sum() / df.visitors.sum()) # 0.0104
Both of the first two lines are one call each and neither raises anything. The only defence is knowing which quantity the decision needs before you write the aggregation.