What Is AB Testing in Data Science? A Hypothesis Test Guide

Learn what AB testing is in data science, how each part maps to a statistical hypothesis test, and how to size, run and read a website experiment.

Published on Updated on 14 min read
Table of contents

Most teams run AB testing by shipping two versions of a page, watching a dashboard and calling a winner when one line pulls ahead. That habit skips the statistics that make the result trustworthy. Without a stated hypothesis, a planned sample size and a fixed stopping point, a "winner" is often noise that happened to look good on the day someone checked.

This article answers three questions. What is AB testing in data science? How does it relate to statistical hypothesis testing? And what does each term, from the null hypothesis to β, look like in a real website experiment? The practical path is to treat every AB test as a hypothesis test: write the hypotheses, pick one metric, fix α, power and sample size in advance, run to the planned stopping point, then report the effect with a confidence interval. The short answer is that AB testing is an application of hypothesis testing to randomized product or website experiments.

What Is AB Testing in Data Science?

AB testing in data science is a randomized controlled experiment that compares two versions of an experience, A and B, by measuring the same outcome for users who were randomly assigned to one version. The goal is a causal claim: the change, and not something else, moved the metric.

Data scientists own this work because the hard parts are statistical. Someone has to define the outcome, size the sample, choose the test, control error rates and interpret the estimate. The tooling that serves the two versions is the easy part.

A quick example makes it concrete. A SaaS site has a signup page. Version A carries the current headline. Version B carries a new one that names the customer's problem. Each visitor sees one headline, and the outcome is whether the visitor signs up. The signup conversion rate for A is compared with the rate for B, and the question is whether the gap is larger than chance alone would produce.

Terms readers confuse with AB testing

  • Split testing: another name for the same method, usually used when the versions are whole pages on different URLs.
  • Multivariate testing: a test that changes several elements at once and measures combinations, where AB testing changes one thing. The multivariate testing guide covers when that is worth the extra traffic.
  • Observational comparison: a comparison of groups you did not assign yourself, which cannot support the same causal claim.

Is AB Testing the Same as Hypothesis Testing?

AB testing is not the same as hypothesis testing. It is an application of it. Hypothesis testing is the general statistical procedure for deciding whether observed data is consistent with a null hypothesis. AB testing uses that procedure inside a specific design: random assignment, a business metric and a decision at the end.

  • What hypothesis testing supplies: the null and alternative hypotheses, a test statistic, a significance level and a way to judge whether a difference is larger than chance.
  • What AB testing adds: random assignment of users to versions, a metric tied to business value, a live-traffic setting where visitors arrive over time, and a ship-or-don't-ship decision.
  • What hypothesis testing covers beyond AB tests: observational comparisons where nobody assigned the groups.

The Cornell data science textbook shows the second case well. It compares birth weights of babies whose mothers smoke with those whose mothers do not, frames the comparison as a hypothesis test, and reports an observed mean difference of about 9.27 ounces. The statistics are the same as in a website test. The conclusion is weaker, because mothers were not randomly assigned to smoke, so the test cannot rule out other differences between the groups.

Practical rule: the statistics of an AB test and an observational comparison look alike, but only random assignment lets you read the difference as an effect of the change.

How Do the Parts of a Hypothesis Test Map to a Website Experiment?

Each part of a hypothesis test has a direct equivalent in a website experiment. The table below uses one running case: a signup page where a new headline is tested against the current one.

Hypothesis-testing termWhat it meansWebsite experiment equivalent
PopulationEveryone the conclusion should apply toAll future visitors to the signup page
SampleThe units actually measuredVisitors who arrive during the test
Null hypothesis (H0)No difference in the outcomeBoth headlines produce the same signup rate
Alternative hypothesis (H1)The difference the test is built to detectThe new headline produces a different signup rate
Test statisticA number summarizing the differenceDifference in conversion rate, scaled by its standard error
α (significance level)Accepted risk of a false positiveRisk of shipping a headline that does nothing
βRisk of a false negativeRisk of discarding a headline that works
Power (1 − β)Chance of detecting a real differenceChance the test finds a real lift in signups
Effect sizeHow large the difference isExtra signups per 100 visitors

Read the table row by row before running a test. If you cannot fill in the right-hand column, the test is not ready to launch. The sections that follow take the rows in turn, in the order you meet them when planning.

The Null and Alternative Hypotheses

The null hypothesis says there is no difference in the outcome between the two versions. The alternative hypothesis states the difference the experiment is built to detect. The test asks whether the data is surprising enough under the null to reject it.

Hypotheses are only useful when they name a change and an outcome. Vague ones, such as "the new page is better", cannot be tested. Here are three concrete pairs, based on the kinds of changes a website optimizer makes: headlines, copy and calls to action.

  • Headline: the null is that a headline naming the visitor's problem and the current headline give the same signup rate. The alternative is that the rates differ.
  • Body copy: the null is that shorter benefit-led copy and the current copy give the same signup rate. The alternative is that the rates differ.
  • Call to action: the null is that "Start free" and "Get started" on the button give the same click-through to signup. The alternative is that the rates differ.

Most teams run two-sided alternatives ("the rates differ") because a change can hurt as well as help. A one-sided alternative ("B is higher") must be chosen before the test, never after seeing which way the data leans. For a longer treatment of writing these statements, see the guide to the AB testing hypothesis.

Control, Treatment and Random Assignment

The control is the current experience, and the treatment is the changed one. Each visitor sees exactly one version, and a random process decides which. That randomness is what makes the comparison causal.

Here is why. Visitors differ in source, device, intent and time of day. If the new headline went only to weekend visitors, any difference in signups would mix the headline's effect with the weekend effect. Random assignment spreads those other factors evenly across both groups on average, so the remaining difference can be attributed to the change.

  • One version per visitor: keep a visitor in the same group on every return visit, so nobody sees both.
  • Assign at the right unit: randomize by visitor, not by page view, or the same person can land in both groups.
  • Run both versions at the same time: never compare this week's control with last week's treatment.
  • Check the split: if the groups should be even and are not, investigate before trusting any result.

The Primary Metric

Choose one primary metric before the test starts, and keep it fixed. For a signup page that is usually the signup conversion rate: signups divided by visitors who saw that version. It is a proportion, so the comparison is a test of two proportions.

Picking the metric afterwards invites false findings. A test usually produces many numbers: clicks, scroll depth, time on page, signups, revenue. If you look at all of them and report whichever one crossed the significance threshold, you have run many hypothesis tests and kept the luckiest. The stated error rate no longer describes what you did.

  • Primary metric: the one the decision rests on, named in advance.
  • Secondary metrics: useful for understanding why the result happened, but not for declaring a winner.
  • Guardrail metrics: numbers that must not get worse, such as refund rate or page load time.

Practical rule: write the primary metric into the test plan before any visitor is assigned, and treat everything else as context.

What Do α, β and Power Mean?

α is the significance level, the accepted risk of a false positive. β is the risk of a false negative, and power is 1 − β, the chance of detecting a real difference. Adobe Target says a commonly used significance level is 5% and a commonly used statistical power is 80%. Use them as common reference points, not as targets.

What does β mean in statistics?

β is the probability of a Type II error: failing to reject the null hypothesis when the alternative is true. In a website test, it is the chance you conclude "no difference" about a headline that really does lift signups. If power is 80%, as in the Adobe Target reference, then β is 20%.

α and β pull in opposite directions for a fixed sample. Lowering the false-positive risk makes the test stricter, which raises the false-negative risk unless you add visitors. The article on examples of Type 1 and Type 2 errors gives more cases.

  • False positive (α): you ship a change that does nothing, and the team believes a lift that is not there.
  • False negative (β): you discard a change that helps, and the gain is never captured.

How do minimum detectable lift and baseline rate drive sample size?

Sample size depends on four inputs: the baseline conversion rate, the minimum detectable lift, α and power. The minimum detectable lift is the smallest improvement worth detecting. Smaller lifts and lower baselines need more visitors, because a small difference is harder to separate from random variation. The guide to the minimum detectable effect explains how to choose it.

You set these inputs before the test. Then a calculator, such as the AB test calculator, turns them into the number of visitors each version needs.

What Is an AB Testing Framework?

An AB testing framework is a repeatable sequence of decisions that turns a business question into a statistically valid experiment and a clear decision. It fixes the outcome, metric, hypotheses, sample size and stopping point before launch, and fixes the analysis before the data comes in.

Steps 1 to 6 happen before the test, step 7 during it, and steps 8 and 9 after.

  1. Outcome: state the business result you want to improve, such as more signups.
  2. Metric: choose the primary metric that measures it.
  3. Hypotheses: write the null and the alternative in terms of that metric.
  4. Versions: define the control and the treatment, changing one thing: one headline, one block of copy or one call to action.
  5. Assignment: set up random assignment so each visitor sees one version, with both versions running at the same time.
  6. Planned sample size and stopping point: fix α, power, baseline and minimum detectable lift, compute the visitor count per version with the AB test calculator, and write down the date or visitor count at which the test ends.
  7. Run: collect data until the planned sample is reached, without stopping on a lucky look.
  8. Analysis: compare the outcomes, estimate the effect and its confidence interval.
  9. Decision: ship, keep the control or iterate, based on effect size and business value. Log the hypotheses, the plan and the outcome together so the next test starts from them.

The value of a framework is that it removes choices from the moment when you can see the data. Every choice made after peeking is a chance to fool yourself.

A Worked Example: Testing a Signup Headline

Planning a headline test shows why sample size is set before the test starts. This example uses the figures from the Adobe Target sample-size guidance. It considers an offer with a 5% baseline conversion rate and a 5% relative lift, which takes the alternative to 5.25%. At 80% power, the example needs 100,000 visitors per offer.

Notice that in the Adobe Target example, a 5% relative lift is a small absolute change: a quarter of a percentage point, from 5.00% to 5.25%. Detecting a gap that small against ordinary visitor-to-visitor variation takes a large sample.

Before: a team writes a new headline for the signup page and sends half the traffic to it. After a few days one version looks ahead, so they stop and ship it. They never wrote down a lift worth detecting or a number of visitors. The "win" may be noise, and a real gain may have been missed.

After: the team writes the null (both headlines give the same signup rate) and the alternative (the rates differ). They fix the baseline, the lift they care about, α and power, and read off the required visitors per version. They run until that number is reached, then analyze once and report the estimated lift with its confidence interval.

What does this mean for a site with little traffic?

If a site gets a few thousand visitors a month, a test sized like this one would take many months. The options are honest ones:

  • Test a bigger change: a larger expected lift needs far fewer visitors than a small tweak.
  • Accept a longer test: the AB test duration calculator converts the sample size into days of traffic.
  • Accept lower power: you then accept a higher chance of missing a real effect, and should say so in the report.

Practical rule: compute the required sample before launch. If you cannot reach it in a reasonable time, change the test, not the stopping rule.

How Do You Read the Result?

Statistical significance answers whether a difference is likely real. It does not say how big the difference is or whether it matters. A p-value below α means the data would be unusual if the null were true. That is all.

The Cornell textbook makes the same point: after the test, it estimates the birth-weight difference and uses a bootstrap confidence interval to show the plausible range around it.

Apply the same habit to a signup test:

  • Report the estimate: give the observed lift in signups per 100 visitors, not only "significant".
  • Give the interval: a wide interval around a positive lift tells you the true gain could be tiny.
  • Judge business value: ask whether the lower end of the interval still justifies shipping.
  • Separate the two questions: "is it real?" is a statistics question, "is it worth it?" is a business one.

A tiny lift can be significant with a huge sample and still not be worth the change. A large lift can fail to reach significance with a small sample and deserve a follow-up test. Effect size and the interval tell these cases apart. The p-value does not.

Where Do AB Tests Go Wrong?

AB tests mostly fail through process errors, not arithmetic ones. Adobe Target notes the common pitfalls: testing with too few visitors, overlooking statistical power and the risk of false negatives, and stopping early after seeing a significant result.

  • Too few visitors: an underpowered test often returns "no significant difference" for a change that works. That is a false negative, and teams read it as proof the idea failed.
  • Peeking and stopping early: Adobe warns that repeatedly checking until significance is reached can underestimate uncertainty and make results unreliable. Each look is another chance for random noise to cross the line.
  • Changing the metric mid-test: switching to whichever metric looks best turns one test into many, and the error rate stops meaning what you set.
  • Changing the stopping rule: extending a test because the result is "almost there" is peeking by another name.
  • Contaminated assignment: visitors who see both versions, or uneven splits, break the logic of randomization.

The fix for all of these is the same: decide in advance, write it down, and follow it.

When Does an Average Result Hide Differences Between Visitors?

A single overall result can hide groups that respond differently. A headline might lift signups among visitors from one source and lower them among visitors from another, and the average lands near zero. The test says "no difference" while two real effects cancel out.

This matters for how you interpret one hypothesis test. The null is about the average across everyone in the test. Failing to reject it does not mean no visitor group was affected. Visitors differ by where they come from, what they searched for and what they expect to find, so one page for all of them is a compromise.

The temptation is to slice the results into many segments after the test and report the one that shows a win. That is the metric-switching problem again. Every segment you check is another hypothesis test, and with enough slices some will cross the threshold by chance.

  • Decide segments in advance: name the visitor groups you expect to differ, with a reason, before launch.
  • Count the comparisons: the more slices you check, the stronger the evidence you should demand.
  • Treat post-hoc findings as hypotheses: a segment that looks promising after the fact is a candidate for its own test, not a result.
  • Size each segment: a segment has fewer visitors, so it needs its own power calculation.

How Does an Automated System Apply the Same Logic?

An automated system applies the same logic as a manual test, only continuously. Polish, our product, is an AI agent for founders without a CRO team. It rewrites headlines, copy and calls to action for each visitor based on where they arrive from (traffic source, new or returning visit, device and country), tests the variations on live traffic and keeps the versions that convert.

Mapped onto the framework, the pieces line up. The variations are the treatments. A conversion, such as a signup, is the outcome. Keeping a version is the decision. The visit signals are what lets the system treat visitors from different origins differently, which speaks to the problem of averages hiding group differences.

Continuous testing changes the statistics, though. A fixed-horizon test looks at the data once, at a planned sample size. A system that tests all the time looks constantly, and that is exactly the situation where peeking inflates false positives. Related approaches, such as the multi-armed bandit, shift traffic toward the better version as data arrives, and they trade some inference quality for faster learning.

What should a data scientist ask of such a system?

Ask these questions of any tool that tests continuously on live traffic and keeps winning versions:

  • How is stopping decided? Look for planned stopping points or a method built for repeated looks, not "stop when it looks significant".
  • How are false positives controlled? Ask what error rate holds across all the versions and looks, not per comparison.
  • How are visitor differences handled? Ask whether segment-level results are separate comparisons with their own error control.
  • What is reported? Ask for estimated effects with intervals, not only a list of winners.
  • Can you audit it? You should be able to see the outcome, the assignment and the decision rule.

Ask the same questions of Polish. You can put it to work on your own signups with a 3-day free trial, then $99 per month, cancel any time.

  • what is ab testing in data science
  • ab testing

Share this post

Your website rewrites itself until it converts.

Polish writes new versions of your headlines and CTAs, tests them on your real visitors and keeps the ones that win.

Start free