A B Testing Hypothesis

Your new landing-page headline is ahead in the dashboard. Someone wants to ship it, sales wants to know whether the leads are any good, and your analyst asks what the team agreed would count as a win. The test is running, but nobody wrote that down.

A good A/B testing hypothesis prevents that conversation. It defines a prediction you can reject, isolates the change you're evaluating, and makes the decision rules explicit before anyone gets attached to a positive result. When you personalize pages for different visitors, that discipline becomes more important, not less.

Why a Hypothesis Is Not Just a Guess

“We think the new headline will convert better” expresses a preference. It doesn't define an experiment.

Better for whom? Does “convert” mean clicking the CTA, submitting a form, or becoming a qualified opportunity? If the headline changes alongside the offer and audience targeting, which change will you credit?

A hypothesis needs to answer those questions before launch. Falsifiable means you can describe evidence that would contradict the prediction, rather than finding a favorable interpretation afterward.

A hypothesis isn't a promise that the treatment will win. It's an agreement about what evidence would justify changing the page.

Replace preference with a testable prediction

Consider this hypothetical replacement: changing a feature-led headline to an outcome-led headline for first-time paid-campaign visitors will increase qualified demo requests by a commercially meaningful amount during the planned measurement window, because those visitors need a clearer connection between the product and their intended outcome.

That statement still needs operational values before launch, but it defines the question. A sufficiently precise result showing no commercially meaningful improvement would challenge the prediction. An inconclusive result would leave it unresolved, not disproved.

• Audience: Identify who is eligible before assignment.
• Change: Describe the treatment precisely enough for someone else to reproduce it.
• Evidence: Define the business outcome and the threshold that matters.

A 2020 academic review of online controlled experiments reported that Google, LinkedIn, and Microsoft each ran more than 20,000 controlled experiments annually, while experiments at Bing, Google, and LinkedIn commonly reached more than 1 million users. The review noted differences in counting practices, so those figures aren't a direct productivity comparison.

The useful lesson isn't to imitate that volume. It's to treat experimentation as a repeatable decision process, with a written question rather than an attractive variation as the starting point.

The Hypothesis Template a Senior Team Uses

Use this working sentence:

Changing [independent variable] for [defined audience] will change [primary metric] by at least [minimum detectable effect] during [measurement window], because [mechanism].

The diagram separates the planning components. The “because” clause connects them to a behavioral explanation.

Make every clause operational

The independent variable should name the actual change, such as replacing a feature description with an outcome-led headline. “Improve the hero section” isn't specific enough. If you change the headline, subheading, and CTA together, call it a messaging-package test rather than claiming you isolated a headline effect.

For the defined audience, use eligibility rules you can evaluate before treatment. Returning mobile visitors from a named campaign could qualify. Visitors who engage with the new headline shouldn't define eligibility, because their engagement occurs after treatment.

Choose a primary metric with an explicit event definition and denominator. Qualified demo requests per randomized eligible visitor is more reproducible than “demo conversion.”

• Minimum detectable effect: Enter the smallest effect you intend the design to detect, based on a meaningful business decision.
• Measurement window: Specify enrollment dates and the time allowed for outcomes to mature.
• Mechanism: Explain why the change should influence behavior, without treating that explanation as established fact.

Separate writing from selection

A copy hypothesis asks whether an alternative message works better for the same eligible population. A targeting hypothesis asks whether selecting messages by visitor context improves outcomes compared with a fixed selection rule.

Don't combine those claims casually. If the treatment gets new copy and new targeting, the experiment evaluates that combined policy. You can make a rollout decision about the package, but you can't claim the targeting alone caused the result.

The template is complete only when its placeholders have actual definitions. A polished sentence with an unspecified audience or measurement window is still an unfinished test plan.

Choosing Metrics That Match the Hypothesis

Suppose a more provocative headline increases CTA clicks, but fewer visitors become qualified leads or book a relevant demo. That's a hypothetical outcome, not a case study, and it illustrates why attention isn't automatically business value.

Choose the primary metric closest to the decision you're making that you can measure reliably within the experiment. For a lead-generation page, qualified demo requests per eligible visitor may be more useful than button clicks. Clicks can remain diagnostic without becoming the basis for rollout.

Primary Metric vs Guardrail Metric for a Landing Page Test

Microsoft documented a Bing headline-display experiment that increased revenue by 12%, reported as more than $100 million annually in the United States at that time, without significantly harming key user-experience metrics in its controlled experimentation paper. That's a historical advertising-interface result, not a forecast for your landing page. The relevant practice is evaluating commercial gains alongside user impact.

Write the trade-off before the result

Record which guardrail movement would block deployment and how much uncertainty you'll tolerate. A non-significant decline doesn't establish that the treatment is harmless, especially when the guardrail is measured imprecisely.

• Quality: Define qualification consistently across variants.
• Experience: Monitor loading and interaction failures.
• Downstream value: Retain activation, retention, or revenue measures even if they mature too slowly to serve as the primary endpoint.

For call-led businesses, Digital Skyrocket's resource on site design that turns traffic into calls offers a relevant design perspective. In the experiment itself, distinguish a call-button interaction from a qualified conversation, and document which one you're evaluating.

Sample Size Without the Statistics Headache

Start with the smallest improvement that would justify maintaining the treatment. Don't start with the uplift someone hopes to put in a presentation.

Your minimum detectable effect determines the sensitivity you're planning for. It isn't the effect the experiment promises to deliver, and you shouldn't redefine it after seeing the treatment's performance.

Use this workflow to organize the inputs.

Translate the business decision into inputs

Use the baseline conversion rate for the eligible audience, not an unrelated sitewide average. Specify the target rate, traffic allocation, significance level, and statistical power in a planning calculator or your experimentation platform.

• Baseline: Match the historical event definition to the planned primary metric.
• Effect: State whether your target is an absolute change or a relative lift.
• Sensitivity: Choose alpha and power before observing outcomes.
• Feasibility: Translate required randomized visitors into an enrollment period using eligible traffic.

The minimum detectable effect guide provides additional context for that planning choice. If the required sample won't fit your operating window, reconsider the audience or the proposed change. Don't inflate the assumed effect just to make the calendar look convenient.

Consider variance reduction carefully

Microsoft's CUPED research reported approximately 50% variance reduction in real Bing experiments, equivalent to roughly doubling traffic or halving the time needed for comparable sensitivity. That result isn't a guarantee for a landing-page test.

CUPED uses pre-experiment behavior to adjust the outcome analysis. It's most useful when that behavior predicts the outcome, and it requires consistent treatment across randomized groups.

Freeze the adjustment specification before reviewing treatment results. For a first-time-visitor audience with little useful history, straightforward analysis may be preferable to a complex adjustment with weak inputs.

Personalization Changes the Hypothesis Game

A single winning headline isn't always the right destination. Visitors from different campaigns may respond differently, but that doesn't mean every apparent subgroup winner deserves its own experience.

The familiar comparison screen is only part of the decision.

Research on personalization through heterogeneous causal effects describes methods for identifying cohorts with different treatment effects and optimizing treatments by cohort. It also highlights complications when eligibility depends on dynamic, stochastic attributes.

Personalization doesn't eliminate randomized testing. It changes what you randomize. You can assign visitors persistently to a fixed experience or to an adaptive selection policy, even when the adaptive group contains different rendered messages.

Decide which claim you need to test

A conditional hypothesis might predict that outcome-led messaging improves qualified-demo conversion for mobile visitors from a particular campaign. A policy-level hypothesis asks whether context-based selection outperforms a fixed message across the eligible population.

• Copy effect: Compare messages within a stable audience definition.
• Selection effect: Compare selection rules using the same candidate messages where practical.
• Combined effect: Evaluate new copy plus adaptive selection as a package, without attributing the result to either component alone.

Examples of website personalization can help you identify candidate contexts. Choose segments because they support a plausible decision, not because a dashboard lets you filter them.

For a supplementary perspective on experimentation, consider the video below.

Make adaptation auditable

Record the selection-policy version, available messages, visitor context, and rendered experience. Specify whether learning continues during evaluation and what information the policy can use, so shared feedback doesn't contaminate the comparison.

When individual cohorts lack adequate evidence, accept the average policy effect as your decision target. Exploratory subgroup findings can inform a fresh validation test, but they shouldn't replace the original hypothesis.

A Full Landing Page Hypothesis Worked End to End

Consider a hypothetical B2B SaaS test for first-time mobile visitors arriving from a paid LinkedIn campaign. The existing headline describes a product benefit. The proposed headline names the customer outcome more directly.

The team's mechanism is that visitors understand the intended result faster and can make a better-informed demo decision. That's an explanation to investigate, not a claim that the team already knows why visitors behave as they do.

Write the preregistration document

Use a compact design specification rather than a creative brief.

The hypothesis sentence then uses the template above, with the team's approved effect threshold and measurement window filled in before launch. Derive that threshold from the value of additional qualified pipeline relative to implementation and maintenance costs.

Next, calculate required sample size from the historical baseline for this eligible audience. There's no defensible numerical estimate without those inputs, so don't borrow another company's conversion rate.

Keep adjacent ideas out of this experiment

If the team also wants a stronger CTA, put it in a separate CTA optimization test plan. Changing it here would turn the experiment into a combined messaging test.

For a separate paid-social workstream, the Rapid Ads growth playbook for Meta is relevant supplementary reading. Don't assume a campaign-specific finding transfers unchanged between acquisition channels.

Before enrollment, save the analysis population, outcome-maturation period, stopping rule, exclusions, and failure criteria. An informal plan describes what the team wants to launch. A preregistration document also describes what would stop the team from trusting or shipping it.

How a Significant Result Can Still Be Wrong

A significant p-value doesn't repair broken assignment, missing conversions, or an analysis rule chosen after seeing the data. It addresses a statistical question under assumptions. Your team still has to establish that those assumptions are credible.

Use the following pitfalls as a review checklist, not as reasons to distrust every experiment.

Freeze decisions when allocation looks wrong

A sample-ratio mismatch, or SRM, means observed assignment counts are inconsistent with the intended allocation beyond expected random variation. Possible causes include routing errors, identity problems, filtering, or missing instrumentation.

Microsoft's guidance on trustworthy experimentation states that SRMs typically invalidate the experiment and make metric movements untrustworthy. Its discussion also covers analysis of 195 Vinted experiments examining SRM detection methods, including daily monitoring versus final-distribution checks.

Don't treat every visible count imbalance as an SRM. Use the planned diagnostic, then freeze rollout decisions if it flags a problem while you investigate assignment and exposure logs. Removing inconvenient users afterward isn't a substitute for finding the cause.

Distinguish monitoring from opportunistic stopping

Looking at operational health during a test is necessary. Repeatedly checking an ordinary fixed-horizon significance test and stopping as soon as it looks favorable changes the false-positive risk.

Use the stopping method you specified: a fixed analysis point or a suitable sequential-testing procedure. A safety stop for a broken form is different from declaring an early effectiveness win.

Decision discipline: You can stop serving a harmful treatment without claiming that an unfinished experiment proved your original hypothesis.

Check durability and data maturity

Novelty can create a temporary response that doesn't persist. Batch processing can delay outcome availability and make an incomplete result look weak. The Microsoft guidance discussed above flags both as interpretation risks.

Inspect treatment effects across enrollment periods and time since exposure, using the durability expectation you wrote before launch. Confirm that outcomes have matured equally across arms before interpreting downstream performance.

For B2B teams, Big Moves Marketing's new approach to B2B attribution provides a useful companion perspective on measurement. Keep the roles distinct: attribution organizes credit across touchpoints, while this experiment evaluates the randomized change.

Paste this into the experiment ticket

Before launch

• Prediction: State the mechanism and the evidence that would challenge it.
• Outcomes: Name the primary metric, denominator, guardrails, and harm tolerances.
• Sensitivity: Justify the detectable effect and calculate the required sample.
• Population: Freeze eligibility, randomization unit, allocation, and exclusions.
• Analysis: Record the stopping rule, maturation period, planned segments, and adjustment methods.
• Personalization: Specify what may adapt and what must remain fixed.

During and after enrollment

• Integrity: Monitor assignment ratios daily and review exposure logging.
• Delivery: Confirm the intended experience renders and events arrive consistently.
• Timing: Review durability and delayed outcomes without rewriting the decision rule.
• Interpretation: Separate planned findings from exploratory subgroup discoveries.
• Decision: Log rollout, rejection, or inconclusive status with uncertainty and guardrail findings.

An invalid experiment and an unsuccessful hypothesis aren't the same thing. With broken implementation, you haven't answered the question; with a valid, precise negative result, you may have answered it well. Keep that distinction in the decision log so the next team doesn't repeat the same test for the wrong reason.

---

Polish offers visitor-level landing-page adaptation, generating headline, subheading, and CTA alternatives and selecting experiences using visitor context. Visit Polish to explore that approach, then define a fixed-control hypothesis that makes your personalization claim testable before you launch.