Explaining A/B testing for UK marketing teams: from hypothesis to results

Two versions of the same page, one job: work out which version deserves to stay. That is the whole idea behind A/B testing, and it is genuinely simple to describe. It gets harder when a test has run for four days, shows a 12% lift, and gets declared a winner before anyone asks whether the numbers can support that claim.

What follows is a practical path through the full cycle — framing a hypothesis, sizing the test, running it without contaminating it, and reading the result honestly.

Start with a hypothesis, not a hunch

"Let's try a green button" is a hunch with a to-do list attached. A hypothesis is a claim you can be wrong about, and it needs a reason behind it.

A useful sentence to fill in: because we observed this behaviour, we believe that this change for this audience will produce this outcome, measured by this metric.

For example: because analytics shows mobile visitors from paid search abandon checkout at the delivery-options step, we believe that showing delivery costs on the product page will increase checkout completions for mobile paid-search visitors, measured by completed orders per unique visitor.

That one sentence forces three decisions early: who you are testing on, what you are changing, and what counts as winning. Teams that skip it tend to spend the analysis phase arguing about which metric should settle the matter.

Designing the test

Change one thing

A test with a new headline, a new hero image and rewritten body copy tells you the combination moved the needle, but not why, and you cannot repeat the trick with confidence. One variable per test keeps the learning transferable. If you have several questions, queue them rather than bundling them.

Define success before you launch

Pick one primary metric and commit to it in writing. Then add guardrail metrics — the things that must not get worse, such as revenue per visitor, refund rate, page load time or support contacts. A variant that lifts newsletter sign-ups while dragging down average order value is not a win.

Decide the audience at the same time. All visitors, or new visitors only? Returning customers behave differently and can swamp a genuine effect from first-time visitors.

Run both versions at the same time

Control and variant should run in parallel, with traffic split randomly at the user level rather than the session level. If someone sees version A on one page and version B on the next, their behaviour muddies the comparison. Sequential tests — variant this week, control next — are not A/B tests; they measure the calendar.

Calculating sample size

Four inputs

Before you launch, you need four things:

• Baseline conversion rate — what your primary metric does now, taken from a reasonably long historical period rather than last Tuesday.
• Minimum detectable effect (MDE) — the smallest lift worth acting on, expressed relative to baseline. Detecting a 20% relative improvement needs far more traffic than detecting a 50% one.
• Significance level — usually 0.05, a 5% tolerance for calling a difference that is not real.
• Power — usually 0.8, meaning an 80% chance of detecting a genuine effect of that size.

Feed those into a sample size calculator and you get a number per variant. Treating it as the total for the whole test is one of the most common arithmetic errors, and it roughly halves the evidence you actually collect.

Then check you can get there

Divide the required sample by your daily eligible traffic. If the answer is six weeks, the test is probably not worth running as designed. Three options: widen the MDE and accept that you can only detect bigger swings, narrow the audience to a segment where the effect may be larger, or choose a metric that fires more often — clicks rather than purchases — while remembering that a proxy metric is still a proxy.

Running the test cleanly

Most damage happens during the test rather than after it. Worth checking:

• Traffic split. A 50/50 test delivering 47/53 has a bug, not a finding. Investigate before you interpret anything.
• Flicker. Redirect-based tools sometimes show the wrong version for a moment. On a slow mobile connection, that changes behaviour.
• Measurement. Make sure your analytics platform and your testing tool agree on what a conversion is.
• Context. Note anything unusual — a price change, an outage, a big campaign launching mid-test — so future-you knows why the numbers look odd.

Seasonality matters too. A test run across a bank holiday weekend or the run-up to a sale tells you something about that period, not about the rest of the year. If it has to run then, record the context and consider repeating it later. And resist opening the dashboard hourly: repeatedly checking results and stopping when they look good inflates the chance of a false positive. Fix the sample size and the end date, then leave it alone.

Reading the result

What a p-value does and doesn't say

A p-value of 0.03 means that if there were genuinely no difference between the two versions, you would see a gap this large about 3% of the time. It does not mean there is a 97% chance the variant is better, and it says nothing about how much money the change is worth.

Confidence intervals tell you more

The interval around your estimated lift shows the range of effects compatible with the data. A variant showing +4% with a confidence interval of -1% to +9% has taught you that the test is inconclusive; the direction is genuinely unclear. A variant showing +4% with an interval of +2% to +6% is a different story altogether.

Statistical significance is not practical significance

With enough traffic, a 0.2% lift can be statistically significant. Whether to ship it depends on the engineering cost, the maintenance burden, and whether the effect survives once novelty wears off. Ask what the lift is worth per month, then decide with that number in front of you.

Segment results with care. Slice a test by device, channel, region, age band and new versus returning, and you are running dozens of comparisons — some will look impressive by chance alone. Choose the segments you care about before the test starts, and treat any other patterns as prompts for the next test rather than conclusions.

Common pitfalls, briefly

• Stopping the moment the variant looks like a winner.
• Editing the variant mid-flight and carrying on as if nothing changed.
• Comparing a variant against last month's performance instead of a live control.
• Testing on a metric that barely fires. If you get eight conversions a week, no design will rescue the analysis.
• Ignoring the novelty effect. Regular visitors may click something new simply because it is new.
• Running overlapping tests on the same audience and treating each result as independent.
• Filing a no-effect result as a failure. Ruling something out is information worth keeping.

Turning a result into a decision

1. Write the outcome in plain language: what changed, for whom, by how much, with what uncertainty.
2. Ship the winner if the lift is real, meaningful, and not funded by a guardrail metric going backwards.
3. Log it. A searchable record of hypothesis, sample, result and decision stops the same idea being retested every eighteen months.
4. Extract the principle, not just the outcome. "Clear delivery pricing beats vague reassurance on mobile checkout" travels much further across a business than "the green button won".
5. Pick the next question. The best test usually comes from the most interesting result, not from a brainstorm.

Keep the discipline tight on design and sample size, and generous with ideas. Most marketing teams do not lack things to test; they lack the patience to test one thing properly, and the nerve to accept an answer they did not want.

Photo: geralt / Pixabay