Years ago I worked on a category-page redesign that won its A/B test on add-to-bag rate and lost on revenue per visit. It was rolled out anyway, because someone in the meeting said “well, the add-to-bag is statistically significant.”
I think about that test often. Not because the decision was wrong (reasonable people could disagree), but because we learned nothing from it. We shipped a change that pushed behaviour in two opposite directions, and we couldn’t tell anyone why.
- +4.1% add-to-bag rate
- −2.3% revenue per visit
There’s a kind of A/B testing that just moves numbers, and a kind that’s genuinely how a team learns about its users. In a testing tool they look identical. They are not the same activity.
A sharp hypothesis is most of the job
If a test is going to teach you something, your guess has to be sharp enough to be wrong in a useful way. I write it in four parts.
- Who and what Which users are doing what today, and why we think they do it.
- The change The specific design decision we're testing.
- The expected result Which number should move, and the reason it should.
- How we'd know we're wrong The warning sign we'd see if our thinking is off, even if the number moves.
That last part is the one most often skipped. It’s also the one that turns a test into research. If you don’t write the warning sign down beforehand, you’ll explain away whatever happens.
The category-page hypothesis was roughly: bigger product images will lead to more add-to-bags, because people can judge fit faster. Reasonable. But it had no warning sign. When revenue dropped, nothing in our thinking predicted it, so we didn’t know what it meant. We should have written: we’re wrong if revenue per visit goes down, because that would mean people are buying quicker, cheaper and less carefully. Then we’d have shipped knowing what we were trading.
Plan how you’ll read the results first
I don’t start drawing a variant until I’ve sketched the dashboard I’d use to read it. Sketching it brings up all the follow-up questions the main number can’t answer.
On a search test I ran in 2024, the main number was how often people who searched went on to buy. The sketch showed we’d also want to split that by:
- Type of search: broad, specific, or a brand name
- Whether they tapped a suggestion or typed the full search
- Whether they changed their search in the same visit
- Time from first letter typed to first result tapped
None of those were the headline. All of them mattered. The test won overall, but the most interesting finding was underneath: people making broad searches bought far more, while people making specific searches did slightly worse. We’d never have seen that from the headline alone, and it shaped the next four releases.
If the only number you can read from the test is the one in the success criteria, your test is a thermostat, not a microscope.
Be honest about how long to run it
I’ve worked on teams where someone peeked on day three, saw a lift, and started writing the launch announcement. I’ve also seen tests run for six weeks “to be sure” when the traffic would have given a clear answer in eleven days. Both are mistakes, and they’re more related than they look.
The habit I try to keep: decide how many visitors you need before the test starts. Base it on the smallest improvement you’d actually act on. If you wouldn’t ship a 0.5% lift, don’t size your test to spot one. You’ll just spend weeks chasing noise.
If 1% more revenue per visit would justify shipping, size the test for 1%. If you need 5% to justify the engineering cost, size it for 5%. Sometimes this tells you the test isn’t worth running at all, because you don’t have enough traffic to see a difference that matters.
Underpowered tests, ones without enough visitors, don’t fail to learn. They confidently teach you the wrong thing.
One more uncomfortable point. If a test runs so long that seasonal peaks, marketing campaigns or stock changes muddy it, discount the result heavily, however clean the numbers look. I’d rather have a rough answer from a clean two weeks than a “significant” one from six weeks that included Black Friday.
Always watch some real people too
A/B tests tell you what happened. They almost never tell you why. If you want to learn, not just measure, you need to watch people alongside every test.
The cheapest version I know: while the test runs, recruit five people who are seeing the new version, using a tool like Maze or a recorded session. Don’t show them the old version. Don’t tell them it’s a test. Watch them do a typical task, then ask what they noticed.
Five sessions won’t change your test, but they will explain its results. On one checkout test, I watched five recordings and saw three people hesitate at the same spot. A small copy change had made the button feel less final. The numbers didn’t say “weak commitment language”. The recordings did. We shipped the new version without that copy change, and got the lift without the drop.
Habits that have saved me
What I want from a test
I want a test that leaves the team with a sharper picture of its users, whichever side wins. If both outcomes would lead to the same shrug, it isn’t worth running.
The category-page test I started with should never have shipped without a follow-up. Knowing which customers were driving the drop in order value would have unlocked the next quarter’s roadmap. Instead we shipped the lift, took the credit and kept moving. I still don’t know what we were trading. That’s the cost of a test that measures without teaching.