A/B test sanity check skill

Stops an agent from calling a winner too early. It checks the sample size, the volume and the traffic split before it looks at the gap, and it is allowed to answer that the test says nothing yet.

Copy it, download it, or let Polish do the same work on your live pages, every day.

Get the skill fileTry Polish free
Category
Testing
Format
SKILL.md
Access
Free with an account

The A/B test sanity check skill file

ab-test-sanity-check/SKILL.md
---
name: ab-test-sanity-check
description: Check whether an A/B test can be called yet, and which version to keep. Use when given conversion numbers for two or more versions of a page.
---

# A/B test sanity check

A conversion rate is a number, not a verdict. Work in this order and stop at the first step that fails.

## 1. Was the sample size set in advance?

If not, say so. A test read whenever someone remembers to look is a test that will eventually show a winner by chance. Compute the size the test needed for the effect it was meant to detect, at 95% confidence and 80% power, and compare it to what ran.

## 2. Is the volume there?
Checking your access

Save it as .claude/skills/ab-test-sanity-check/SKILL.md in your project. Your agent loads it on its own when a task matches the description at the top of the file.

About this skill

What is the A/B test sanity check skill?

What it does

  • A check that the sample size was decided before the test ran
  • Volume guards on visitors and on conversions
  • A look at the traffic split, where delivery problems hide
  • A two-sided two-proportion z-test, reported with the numbers
  • The list of things that invalidate a test
  • A verdict in one sentence, including the honest not yet

When to use it

  • Reading a test someone else started
  • Deciding whether to ship a variant
  • Sizing a test before launching it
  • Settling an argument about a result that looks too good

More skills