A/B Testing

A/B testing is the randomized experiment that decides whether a model, prompt, or feature change truly improves a live metric — or just moved on noise.

Published Updated

On this page

Definition

A/B testing is a randomized controlled experiment run on live traffic: you split users at random into a control group that sees the current version (A) and a treatment group that sees the change (B), then compare a chosen metric between the two and run a statistical test to decide whether the difference is real or just noise. The random assignment is the whole point — because the only systematic difference between the groups is the change itself, any reliable gap in the metric can be attributed to that change rather than to who happened to show up that day.

It matters because most changes do not help, and the ones that hurt are invisible without a control. Teams that run experiments at industrial scale report the same uncomfortable number: at Microsoft roughly a third of ideas improve the metric they were designed to move, and at several other companies the rate is lower still (Kohavi and colleagues, Trustworthy Online Controlled Experiments, 2020). If two-thirds of your "improvements" are flat or negative, shipping on intuition means shipping regressions you never see. For an AI product, "the change" is usually a new model version, a rewritten prompt, a different retrieval strategy, or a tuned temperature — and the reason to A/B test it is that a change which looks better in the lab routinely fails to move any outcome a user cares about.

How It Works

An A/B test has three moving parts: random assignment, a metric, and a significance test. Get any of the three wrong and the result is worthless in a way no dashboard will warn you about.

Random assignment means each unit — usually a user, sometimes a session or a request — is deterministically hashed to a bucket so that the same user always sees the same variant, and the split between variants is uncorrelated with anything about the user. This is what lets you treat the two groups as otherwise identical. If instead you ship the new prompt to everyone on Tuesday and compare against Monday, day-of-week, news events, and traffic mix all confound the result; that is a before/after comparison, not an A/B test.

A metric is the single number you decided, before running, would define success — conversion rate, task-completion rate, thumbs-up rate on an assistant, seven-day retention. Choosing it up front is not bureaucracy: if you pick the metric after seeing the data, you will find some metric that moved, because with enough metrics one always does.

A significance test answers whether the observed gap is larger than what random splitting alone would produce. Suppose a prompt change is tested for a week. The control prompt gets 10,000 users and 2,000 of them complete the task (20.0%); the treatment prompt gets 10,000 users and 2,200 complete it (22.0%). The observed lift is 2 percentage points, a 10% relative improvement — but is it real? The standard tool is a two-proportion z-test. You pool the two rates to estimate the noise you would expect under "no true difference":

pooled p = (2000 + 2200) / (10000 + 10000) = 0.2100
SE       = sqrt( 0.21 * 0.79 * (1/10000 + 1/10000) ) = 0.00576
z        = (0.22 - 0.20) / 0.00576 = 3.472
two-sided p-value = 0.00052

A p-value of 0.0005 means that if the two prompts were truly identical, you would see a gap this large by chance about 1 time in 1,900. That is well under the conventional 0.05 threshold, so you reject "no difference" and conclude the lift is real. Note what the p-value does not say: it is not the probability the change helps, and it says nothing about whether 2 points is worth the engineering cost. Statistical significance and practical significance are different questions, and only the second decides whether you ship.

The mirror image of the significance test is the sample-size calculation you should run before the experiment. You cannot detect an effect you were never powered to see, and a test that ends "not significant" on too few users has told you nothing. To detect a 2-point lift on a 20% baseline at 80% power and 5% significance:

n per arm = (1.96 + 0.84)^2 * ( 0.20*0.80 + 0.22*0.78 ) / (0.02)^2
          = 7.84 * 0.3316 / 0.0004
          = 6500

About 6,500 users per arm, 13,000 total. Because the effect size sits in the denominator squared, detecting a lift half as large (1 point) needs roughly four times the traffic. This is why small teams and low-traffic features often cannot A/B test their way to an answer on subtle changes — the experiment would take months to reach the sample it needs.

Real-World Applications

Large online products run A/B testing as standing infrastructure, not as one-off studies. Microsoft's experimentation platform (ExP), Booking.com, Netflix, LinkedIn, and Airbnb each run hundreds to thousands of concurrent controlled experiments; Booking.com has publicly described operating over a thousand at once across its site. Every visible change — a ranking tweak, a layout, a recommendation model, an onboarding flow — passes through a randomized comparison before it becomes the default, precisely because the base rate of ideas that work is so low that intuition is not trusted.

For AI systems specifically, A/B testing is how model and prompt changes are actually validated in production, downstream of any offline check. A new search-ranking or recommendation model is typically evaluated offline first (on historical data or a held-out set), but it is not launched on the strength of that score — it is put behind an online A/B test, often using interleaving, because offline metrics regularly fail to predict the online outcome. The same discipline applies to LLM features: a rewritten system prompt, a switch from one model version to another, a change to retrieval or temperature, or a new safety filter is routed to a random slice of live traffic and judged on a real behavioral metric such as task completion, follow-up rate, or explicit user feedback. This is the concrete link between offline evaluation and shipping: a Benchmark or an LLM-as-a-Judge score is a filter that decides which candidates are worth testing online, while the A/B test on Production Systems decides which one actually ships.

Key Concepts

Minimum detectable effect (MDE) is the smallest true lift you want the experiment to be able to catch. It is a business decision disguised as a statistics term: setting it too small demands unreachable sample sizes, while setting it too large means you will call a real-but-modest improvement "not significant" and discard it.

Statistical power is the probability the test detects a real effect of the MDE's size. The convention is 80%, which means that even a correctly designed test misses a genuine effect one time in five. Power, MDE, sample size, and significance level are four dials on one instrument — fix any three and the fourth is determined, which is why the sample-size formula above and the significance test are the same equation read in opposite directions.

Guardrail metrics are the numbers you monitor to make sure a win on your target metric did not come at an unacceptable cost elsewhere. For an AI feature the usual guardrails are latency, cost per request, error and timeout rates, and safety measures such as refusal or toxicity rates. A prompt that raises helpfulness while doubling median latency has probably not earned its launch.

The offline–online gap is the recurring finding that a change which improves an offline score does not reliably improve the live metric. The causes are specific: the offline evaluation set does not match the distribution of real traffic; the change adds latency that offline scoring ignores; or the metric being optimized offline (say, a judge's rating of answer quality) is only loosely correlated with what users actually do. A/B testing exists to close this gap, and treating a good offline number as permission to skip it is the most common way AI teams ship regressions.

Challenges

The arithmetic of a single, correctly-run test is the easy part. The failures that matter are the ones that produce a confident, significant-looking result that is wrong.

  • Peeking and early stopping. The p-value's 5% guarantee holds only if you decide the sample size in advance and look once. If you monitor the test daily and stop as soon as it crosses p < 0.05, you have given yourself many chances to cross the line by luck, and the real false-positive rate climbs far above 5%. Simulating repeated A/A tests — two identical variants, where every "win" is by definition a false positive — makes the inflation concrete: checking once gives the expected 5.0%, but checking 5 times gives 14.1% and checking 10 times gives 18.7% (values reproduced by the code below, and matching the classic group-sequential result of Armitage, McPherson and Rowe, 1969). The fix is to fix the sample size ahead of time, or to use a test built for continuous monitoring — sequential testing or always-valid ("anytime-valid") confidence sequences.

  • Novelty and primacy effects. A change to a UI or an assistant's behavior can move a metric simply because it is new: users click the unfamiliar thing, or reject it out of habit, and the effect decays over days or weeks. A one-week test that captures only the novelty spike will overstate a lasting benefit. The defense is to run long enough to see the metric settle, and to check whether the effect on returning users differs from the effect on newcomers.

  • The offline–online gap, again. Because it is the single most common trap for AI changes, it belongs on any list of pitfalls: a prompt that scores higher with an LLM judge, or a model that beats the incumbent on a benchmark, can still lose the A/B test on latency, distribution shift, or a metric the offline eval never captured. The offline win is a hypothesis, not a result.

  • Interference between units. The math assumes one user's treatment does not affect another's outcome. That breaks in marketplaces (a treatment that helps buyers can starve the control group of inventory), in social features, and in shared-resource systems where a heavier model in the treatment arm slows the control arm through contention. When units interact, standard randomization understates or inverts the true effect, and cluster-level randomization is needed instead.

  • Multiple comparisons. Watching twenty metrics at α = 0.05 means you expect one false positive even when nothing changed. Pre-register the primary metric, and correct the threshold (or treat the rest as guardrails, not wins) for everything else.

Code Example

The full calculation from How It Works — the two-proportion z-test, the sample-size formula, and the peeking simulation that produced the 5.0% / 14.1% / 18.7% figures above — in one self-contained script. Run as published, its output is shown beneath it.

import math, random

# --- Two-proportion z-test on the worked example ---
n_A = n_B = 10_000
succ_A, succ_B = 2_000, 2_200          # 20.0% vs 22.0%
p_A, p_B = succ_A / n_A, succ_B / n_B
p_pool = (succ_A + succ_B) / (n_A + n_B)
se = math.sqrt(p_pool * (1 - p_pool) * (1/n_A + 1/n_B))
z = (p_B - p_A) / se
p_value = math.erfc(abs(z) / math.sqrt(2))     # two-sided

print(f"control:   {p_A:.1%}   treatment: {p_B:.1%}   lift: {p_B-p_A:+.1%}")
print(f"pooled p:  {p_pool:.4f}   SE: {se:.5f}")
print(f"z = {z:.3f}   two-sided p = {p_value:.5f}")
print("significant at 0.05?", p_value < 0.05)

# --- Sample size needed to detect a 2pp lift at 80% power ---
z_alpha, z_beta = 1.96, 0.84            # 5% two-sided, 80% power
base, mde = 0.20, 0.02
pt = base + mde
n_per_arm = (z_alpha + z_beta)**2 * (base*(1-base) + pt*(1-pt)) / mde**2
print(f"\nsample size per arm (2pp lift, 80% power): {math.ceil(n_per_arm)}")

# --- Peeking: false-positive rate when you stop early on p < 0.05 ---
random.seed(42)
def one_experiment(peeks, per_peek=1000, true_rate=0.20):
    a = b = sa = sb = 0
    for _ in range(peeks):
        for _ in range(per_peek):
            a += 1; sa += random.random() < true_rate
            b += 1; sb += random.random() < true_rate
        pa, pb = sa/a, sb/b
        pp = (sa + sb) / (a + b)
        s = math.sqrt(pp * (1 - pp) * (1/a + 1/b))
        if s > 0:
            zz = (pb - pa) / s
            if math.erfc(abs(zz) / math.sqrt(2)) < 0.05:
                return True             # stopped early and declared a "winner"
    return False

for peeks in (1, 5, 10):
    trials = 20_000
    fp = sum(one_experiment(peeks) for _ in range(trials))
    print(f"peeks={peeks:2d}  false-positive rate = {fp/trials:.3f}")

Output:

control:   20.0%   treatment: 22.0%   lift: +2.0%
pooled p:  0.2100   SE: 0.00576
z = 3.472   two-sided p = 0.00052
significant at 0.05? True

sample size per arm (2pp lift, 80% power): 6500
peeks= 1  false-positive rate = 0.050
peeks= 5  false-positive rate = 0.141
peeks=10  false-positive rate = 0.187

The z-test confirms the 2-point lift is real (p = 0.0005), the sample-size line shows the ~6,500-per-arm this design required, and the peeking block is the warning that gives the whole method its teeth: the same 5% test, checked ten times and stopped on the first win, is wrong nearly one time in five.

Frequently Asked Questions

Offline evaluation scores a change on a fixed dataset — a benchmark, a held-out set, an LLM-as-a-judge run — before it ever meets a user. A/B testing measures the change on live traffic against a concurrent control group. A change can win offline and lose online, so the A/B test is the arbiter for what actually ships.
It depends on your baseline rate and the smallest lift worth detecting (the minimum detectable effect). Detecting a 2-percentage-point lift on a 20% baseline metric at 80% power and 5% significance needs about 6,500 users per arm. Smaller effects and rarer events need dramatically more — halving the effect roughly quadruples the sample.
Every time you check for significance and reserve the right to stop when p < 0.05, you get another chance to cross the line by luck. Checking ten times inflates the real false-positive rate from 5% to roughly 19%. Either fix the sample size in advance, or use a test designed to be valid under continuous monitoring.
Yes, and it is often the only trustworthy way to validate one. Route a random fraction of traffic to the new prompt, keep the rest on the old one, and compare a real outcome — task completion, thumbs-up rate, retention — with a significance test. Offline judge scores are a useful filter but do not guarantee an online win.
A guardrail is a metric you do not want to harm even if your target metric improves — latency, error rate, cost per request, or a safety measure. A prompt that lifts helpfulness but doubles latency or raises refusals may fail on guardrails, so a good A/B test watches several metrics at once, not just the one it is trying to move.

Related Terms

LLM-as-a-Judge

Using one model to grade another's output against a rubric: how it works, the biases that break it, and the human agreement rate you can actually reach.

Benchmark

An AI benchmark is a standardized test or dataset used to evaluate and compare the performance of different AI models across tasks like reasoning and coding.

Precision and Recall

Precision, recall, F1 and the confusion matrix worked through one imbalanced example — and why the model with the best accuracy can catch nothing at all.

Production Systems

A model serving live users under latency and uptime guarantees — judged at the 99th percentile under real load, not by the median on a laptop.

Monitoring

Why a deployed model rots while every dashboard stays green — and what you can actually measure when the true labels arrive weeks late, or never.

Continue Learning

Explore our use-case guides and prompts to deepen your AI knowledge.

Browse AI Tools← Back to Glossary