AB Testing: What It Is and How to Use It

Learn what AB testing is, how a test works, how to check statistical significance, and how to run a test on your site without fooling yourself.

Published on Updated on 13 min read
Table of contents

Most teams handle AB testing in one of two ways. They change a headline because someone senior prefers it, or they run a quick split and declare a winner the moment one version pulls ahead. Both skip the parts that make a result trustworthy: a clear question, a fixed metric, enough traffic and a fair comparison. The result is a decision that feels evidence-based and is really a guess.

What is AB testing and how does it work? How long should you run one? And how do you know a lift is real? The short answer: you split visitors at random between the current version and one changed version, measure one pre-chosen metric, and compare the groups once the planned sample and duration are done. This guide lays out each step, the numbers that are rules of thumb, and where the method stops being enough.

What Is A/B Testing?

A/B testing (also written AB testing or split testing) is a method for comparing two versions of one thing, such as a web page, an email or a button, to see which gets more of a desired action. Version A is the current one. Version B has a deliberate change. Visitors are randomly assigned to one version, and the difference in results is credited to the change. The data decides, not opinion.

On a website, teams test the elements that sit between a visitor and a signup:

  • Headlines: the first line a visitor reads, which sets whether they keep reading. See headline testing for how to approach it.
  • Copy: the supporting text that explains the offer and handles objections.
  • Calls to action: button wording, placement and prominence.
  • Forms: the number and type of fields between interest and completed signup.

Four terms come up throughout this guide:

  • Control: the current version, the baseline you compare against.
  • Variant: the version with one deliberate change.
  • Conversion: the action you want a visitor to take, such as completing a signup.
  • Hypothesis: a specific prediction of what will change, which metric will move and why.

Related terms people confuse with it: multivariate testing changes several elements at once and measures combinations, while A/B testing changes one. Multivariate testing needs far more traffic for that reason. A fake door test measures interest in something that does not exist yet, while an A/B test compares two things that do.

How Does an A/B Test Work?

An AB test works in four moves: traffic is split, each group sees one version, behavior is measured, and the results are compared. The logic is simple. Because the groups are alike and only one thing differs, a gap in outcomes points to that one thing.

  1. Split the traffic. Incoming visitors are divided between control and variant, usually evenly.
  2. Serve one version to each group. A visitor sees the same version on every visit during the test.
  3. Measure behavior. You record how many in each group complete the conversion.
  4. Compare results. You check whether the gap is larger than chance alone would plausibly produce.

Why only one thing should change

If you change the headline, the button color and the form together, a lift tells you nothing about which change caused it. You cannot repeat the win or learn from it. Flagsmith's guide to doing A/B testing properly lists changing several things at once among the common mistakes. One change per test keeps the result readable.

Practical rule: one test, one change, one primary metric.

Control and Variant: What Are You Actually Comparing?

The control is the version you run today, and the variant is a copy with exactly one deliberate change. Everything else must stay the same: the traffic sources, the time period, the page speed and the surrounding content.

That last point is easy to break. If the variant loads slower than the control, or you launch it during a promotion that the control never saw, you are measuring those differences too. Both versions should run at the same time, on the same kind of traffic.

  • Control: leave it untouched for the whole test, so the baseline is stable.
  • Variant: change one element and write down exactly what changed.
  • Conditions: run both at the same time, not one after the other.
  • Tracking: confirm the conversion is recorded identically in both versions.

Running one version this week and another next week is not an A/B test. It is a before and after comparison, and anything that changed between the two weeks contaminates it.

Why Does Random Assignment Matter?

Random assignment keeps the two groups comparable, so the difference in results can be credited to the change and not to who happened to see it. If returning customers mostly saw the variant and new visitors mostly saw the control, you would be comparing audiences, not versions.

Random does not mean "whoever arrives first" or "visitors from one channel." It means each visitor has the same chance of landing in either group, and stays there.

Check for sample-ratio mismatch

If you planned a 50/50 split and the groups come out noticeably uneven, something is wrong with assignment, tracking or the page itself. This is called a sample-ratio mismatch. The methods described in the Flagsmith and Statsig guides include checking for it before trusting a result. A broken split means the result is unreliable, however good the lift looks.

Is AB Testing Worth It?

AB testing is worth it when you have a real problem, enough traffic to reach a clear answer and a single metric that matters. In that situation it replaces opinion with evidence about what your visitors respond to. Outside it, the cost can exceed the benefit.

The costs are real:

  • Traffic: every visitor sent to a weaker variant is a visitor who may not convert.
  • Time: a valid test runs for a planned period, not until you feel sure.
  • Discipline: you must fix the metric and duration in advance and hold to them.
  • Waiting: low-traffic sites may wait a long time for a clear result, because small samples cannot separate a real effect from noise.

The benefit is that decisions stop depending on who argues best. Teams also build a record of what did and did not move the metric, which compounds over time.

Practical rule: test when you have a drop-off worth fixing and traffic to settle it. If you have neither, fix obvious problems directly and save testing for later.

How Do You Run an A/B Test, Step by Step?

The accessible guides, including Flagsmith and Statsig, converge on one order of work. Follow it from the problem to the written result.

  1. Find a real drop-off using existing data.
  2. Write a hypothesis that names the change, the metric and the reason.
  3. Choose one primary metric before you look at any results.
  4. Estimate sample size and duration from the effect worth detecting.
  5. QA both versions so each works on every device you care about.
  6. Run the test for the planned period without stopping early.
  7. Check significance and the split, then review secondary metrics.
  8. Roll out a supported winner, or document the result and iterate.

The next four sections take these in groups.

Start From a Real Drop-Off, Not a Preference

A good test starts with evidence that visitors leave at a specific point, not with an idea someone likes. Look at where people drop off: a landing page with strong traffic and weak signups, a form that many start and few finish, a pricing page with high exits.

  • Analytics: find the step with the biggest fall between visitors and conversions.
  • Session behavior: watch where people stop scrolling or abandon a form.
  • Message match: check whether the page repeats the promise of the ad or link that brought the visitor. A conversion rate optimization audit walks through this kind of review.
  • Your own question: write down what you think is wrong before you invent a fix.

Testing a button color because a colleague dislikes it is a preference. Testing a form because 60 percent of starters abandon it in your own analytics is a problem. Only the second justifies the traffic.

Write a Hypothesis and Pick One Primary Metric

A hypothesis states what you will change, which metric should move and why. "Make the page better" is not one. "Cutting the form from six fields to three will raise completed signups because less effort lowers abandonment" is. For more on writing them, see this guide to the A/B testing hypothesis.

A worked hypothetical: six fields to three

This is a hypothetical, not a case study, and it follows the signup-form idea described in Flagsmith's product management guide. The numbers below are invented only to show the flow and are not research findings.

Before: a SaaS signup page asks for six fields. Analytics show many visitors start the form and leave before finishing. The team suspects length is the cause.

Hypothesis: reducing the form from six fields to three will increase completed signups, because fewer fields mean less effort at the moment of commitment.

Setup:

  • Control: the six-field form, unchanged.
  • Variant: the same page with three fields. Nothing else differs.
  • Primary metric: completed signups per visitor, chosen now.
  • Secondary metrics: signup quality, such as whether new accounts go on to activate, so a lift in volume that brings worse leads is visible.
  • Plan: a fixed duration and sample size, set before launch.

After: the test runs for the planned period. The team checks that the split matches the plan, then reads the primary metric. If the three-field version shows a gap that passes the significance check and activation does not fall, they roll it out and write down the result. If the gap is within noise, they record that fewer fields did not clearly help and move on to the next hypothesis, perhaps the headline.

Fix the metric first

Choose the primary metric before you look at results. Deciding afterward lets you pick whichever number looks best, which almost guarantees a false win. For forms, see website forms for lead generation for what to measure.

Practical rule: one hypothesis, one primary metric, written down before launch.

How Much Traffic and Time Does a Test Need?

Sample size depends on the effect worth detecting: the smaller the lift you want to catch, the more visitors you need. Duration then follows from how long it takes your traffic to reach that sample. The minimum detectable effect is the input that drives both.

Publishers give rules of thumb, and they differ because contexts differ:

  • At least a week: Flagsmith recommends running a test for at least a week to capture weekday and weekend behavior, according to Flagsmith. It adds that B2B or longer-consideration purchases may need several weeks.
  • 1 to 3 weeks: Twilio says a test should run for at least 1 to 3 weeks, depending on the sample size needed for significance, according to Twilio.
  • 100 to 500 contacts: BenchmarkONE recommends at least 100 contacts, and 100 to 500 contacts per test, for email testing, according to BenchmarkONE.

Use these figures as reference points to compare against, never as targets. The email numbers come from an email context and do not transfer to a website test. BenchmarkONE's advice to review results around days 5 to 7 is also specific to email.

To turn your own baseline and expected lift into a real number, use the A/B test duration calculator.

Practical rule: a publisher's duration is a floor to sanity-check against, not a plan. Your plan comes from your traffic and the effect you want to detect.

QA, Run the Test and Read the Result

Before launch, check that both versions work on the devices and browsers your visitors use, and that conversions are recorded in each. A variant that breaks on mobile will lose for the wrong reason.

Then run the test for the planned period.

  • Do not stop early: an apparent lead on day two is often noise that fades.
  • Do not peek and decide: looking is fine, acting on an early gap is not.
  • Check secondary metrics: confirm the winner did not hurt something else, such as signup quality.
  • Check the split: confirm the groups match the planned ratio.

At the end, roll out a supported winner. If the result is unclear, document what you tested, what happened and what you will try next. A flat result is information too. The same discipline applies to landing page split tests.

How Do You Calculate Statistical Significance in an A/B Test?

Statistical significance tells you how unlikely the gap between versions would be if they actually performed the same. A small p-value means the gap would rarely appear by chance alone. You calculate it from the number of visitors and conversions in each group, which a calculator does for you.

The common rule of thumb is a 5% p-value, which corresponds to a 95% confidence level, according to Twilio. In plain words: if the versions were truly equal, you would see a gap this large less than 5 times in 100.

To check your own result:

  1. Enter visitors and conversions for the control.
  2. Enter visitors and conversions for the variant.
  3. Read the reported significance or p-value.
  4. Compare it with the threshold you set before the test.

The A/B test calculator does this for you. Statsig also covers an alternative approach in its Bayesian A/B test glossary entry.

What significance does not tell you

  • Size of the effect: a significant result can be tiny and not worth shipping.
  • Why it happened: it shows a difference, not the cause.
  • Certainty: at 95% confidence, some wins are still false. See examples of type 1 and 2 errors.
  • Universality: 5% and 95% are a rule of thumb, as Twilio calls them, not a law. A team with costly decisions may demand more.

Practical rule: set your threshold before the test, and treat a passing result as evidence, not proof.

What Mistakes Ruin A/B Tests?

Four mistakes appear across the guides, and each produces a result that looks solid and is not.

  • Changing several things at once: you cannot tell which change worked, so you cannot learn from it.
  • Choosing the metric after seeing data: you will find some number that moved, and call it the goal.
  • Stopping too early on an apparent lift: early gaps often shrink as the sample grows.
  • Treating a lift as meaningful without statistical evidence: a higher number is not a proven difference.

Ronny Kohavi, former Vice President and Technical Fellow at Airbnb, gives a short rule for a lift that looks too large: "when something looks too good to be true, investigate," as quoted in Antoine Buteau's notes on his Lenny's Podcast interview.

Two more are worth avoiding. Copying publisher rules as laws: a week from Flagsmith, the 95% confidence level from Twilio or 100 contacts from BenchmarkONE are reference points from specific contexts, not rules for your site. Ignoring a broken split: an uneven ratio means the result cannot be trusted.

The fix for all of them is the same: decide the question, metric, duration and threshold before launch, then follow the plan. More worked scenarios appear in these A/B testing examples.

Where Does Classic A/B Testing Fall Short?

Classic A/B testing crowns one version for everyone. It finds the version that works best for the average visitor in your sample. But visitors arrive from different places with different intent, and one page cannot match all of them. A headline that wins on average may lose with the people who matter most.

Polish approaches this differently. It is an AI agent that reads visit signals to rewrite headlines, copy and calls to action for each visitor, tests variations on live traffic and keeps the versions that generate conversions. Instead of picking one winner for everyone, it adapts what each visitor sees. This ties to the wider idea of website personalization.

To be fair about the evidence: the guides cited here describe controlled comparisons of versions. They say nothing about adapting content per visitor, so the case for that approach rests on the logic above, not on those guides. Classic testing remains the right tool for a single, clean question such as the form-length test earlier. For a side-by-side view, see Polish versus manual A/B testing.

What Does A/B Testing Software Do?

A/B testing software splits audiences, serves variants and collects interaction data, according to Salesforce. That removes the manual work of assignment and tracking, and it keeps each visitor in one group.

When you evaluate any tool, look for:

  • Reliable random assignment: with a way to spot a broken split.
  • Clear metric setup: so you fix the primary metric before launch.
  • Significance reporting: with the method stated, frequentist or Bayesian.
  • Secondary metrics: to catch unintended effects.
  • Fit for your traffic: low-traffic sites need realistic expectations about test length.

Pricing and plans change often, so check each vendor directly. For a broader overview, read this roundup of A/B testing tools.

Your First A/B Test: What to Have Ready

The eight steps above are the process. Before your first launch, have three things ready that the steps leave implicit:

  • A written threshold: decide the significance level before launch and do not move it afterward.
  • One or two secondary metrics: such as signup quality, to catch a win that brings worse leads.
  • The two calculators: the duration calculator to size the test and the A/B test calculator to read it.

AB testing works when each piece supports the next. A real drop-off gives you a hypothesis, the hypothesis fixes the metric, the metric and expected effect set the sample and duration, and the significance check tells you whether to trust the gap. Skip one and the rest lose their value. The rules of thumb in this guide are starting points, and your traffic and sales cycle decide the rest.

Polish is an AI agent for founders and website owners who want more signups. It reads visit signals to rewrite headlines, copy and calls to action for each visitor, and keeps testing variations on live traffic to retain the versions that convert. See how it works at Polish.

  • ab testing
  • conversion optimization

Share this post

Your website rewrites itself until it converts.

Polish writes new versions of your headlines and CTAs, tests them on your real visitors and keeps the ones that win.

Start free