DeltaV Digital
Back to Glossary

A/B Testing

A/B testing (also called split testing) is a controlled experiment that randomly divides traffic between two versions of a webpage, email, ad, or other digital asset to determine which one performs better against a specific, measurable goal.

What A/B Testing Means in Practice

A/B testing is one of the most direct ways to answer the question that matters in marketing: “Does this change actually improve results?” Instead of debating whether a new headline, button color, or page layout is “better,” you run both versions simultaneously with real traffic and let the data decide. The concept is simple. The execution is where most teams run into trouble.

The mechanics are straightforward. You create two versions of something: a control (the original, version A) and a variant (the change you want to test, version B). Traffic is randomly split between the two versions, with each visitor seeing only one. After enough data has accumulated to reach statistical significance, you compare the results against your target metric, typically conversion rate, click-through rate, or revenue per visitor, and declare a winner, or determine that the difference isn’t meaningful enough to act on.

A/B Testing vs. Split Testing: Same Method, Two Names

“A/B testing” and “split testing” describe the same experiment. The names come from the two halves of the method: “A/B” refers to the two versions being compared, and “split” refers to dividing the audience between them. Marketers, testing platforms, and ad networks use the terms interchangeably, so a split test in Meta Ads Manager, a subject line test in your email platform, and an A/B test on a landing page are all the same controlled comparison.

When you test more than two versions at once (A vs. B vs. C), the experiment is often called an A/B/n test. It’s still a split test. It just divides traffic across more variants, so each one needs more time to collect a valid sample.

Split URL Testing: The One Nuance Worth Knowing

The one place the terms diverge is split URL testing. In a standard A/B test, the testing tool modifies elements on a single URL, swapping a headline or form for half of visitors. In a split URL test, each variant lives at its own URL (for example, /pricing/ and /pricing-v2/), and the tool redirects a share of visitors to the alternate page. Some platforms shorten this to “split testing,” which is why you’ll occasionally see the terms treated as different things.

Split URL testing is the better choice when the variant is a full redesign, a new page template, or a change to backend functionality that can’t be injected with a script. It also carries SEO considerations that on-page tests don’t. Google Search Central recommends pointing a canonical tag on each variant URL back to the original, using temporary 302 redirects rather than permanent 301s, showing Googlebot the same content users see, and ending the test once you have a result.

A/B Testing vs. Multivariate Testing

One misconception worth addressing early: A/B testing isn’t the same as multivariate testing. An A/B test changes one element at a time (or tests two distinct page designs against each other). Multivariate testing evaluates multiple variables and their combinations simultaneously, such as three headlines crossed with two hero images, which produces six combinations. That design can reveal interaction effects between elements, but it divides traffic across every combination, so it requires significantly more traffic to reach valid conclusions.

For most businesses, A/B testing is the more practical starting point because the traffic requirements are lower and the results are easier to interpret. Multivariate testing earns its place on high-traffic pages where you’ve already found the big wins and want to tune how several elements work together.

What to Test

What separates productive A/B testing from wasted effort is what you choose to test and how you structure the experiment. Testing a button color in isolation is a common example in blog posts, but it’s rarely where real performance gains live. The tests that move the needle tend to target elements tied directly to the conversion rate: headline copy on a landing page, the structure of a form, the placement and language of a call to action, the trust signals that address a visitor’s hesitation, or the sequence of information on a service page.

In practice, A/B testing spans every channel. Email marketers test subject lines, send times, and body content. Paid media teams test ad copy, creative assets, and audience targeting configurations. Web teams test page layouts, navigation patterns, and checkout flows. The principle is always the same: isolate one variable, measure the impact, and make decisions based on evidence rather than opinion.

For multi-location businesses, A/B testing introduces an additional layer of complexity. A headline that converts well for a dermatology practice in Dallas might underperform for the same brand in Minneapolis because local audiences describe their problem differently, face different competitors, or care about different insurance networks. We see this regularly across healthcare and professional services clients: location-level performance variation means that a “winner” at the aggregate level can mask underperformance in specific markets. The best testing programs account for this by segmenting results geographically when sample sizes allow it.

Why A/B Testing Matters for Your Marketing

A/B testing is the mechanism that turns your website, email program, and ad campaigns from static assets into continuously improving systems. Without it, you’re making changes based on intuition, internal consensus, or what a competitor did. With it, you’re making changes based on what your actual audience responds to.

The business impact compounds over time. A single test that lifts conversions by 10% might seem incremental. But when you run 10 to 15 tests per quarter across your highest-traffic pages, those gains stack. Harvard Business Review research on online experiments found that companies with mature testing cultures consistently outperform competitors because they make hundreds of evidence-based improvements that compound rather than relying on a few large, risky redesigns. The same research reports that only about 10% to 20% of experiments at Google and Bing produce positive results, which is exactly why testing beats intuition: most ideas that seem obviously better aren’t.

For organizations managing marketing budgets across SEO, paid media, and web, A/B testing also reveals where to allocate spend more effectively. If a landing page test lifts conversion rate from 4% to 6%, the same traffic now produces 50% more leads, and that improvement amplifies every dollar spent driving visitors to that page, whether they come from organic search, PPC, or email. Testing doesn’t just improve individual pages. It improves the ROI of your entire acquisition system.

Testing also reduces the risk of expensive mistakes. A full site redesign that launches without testing is a gamble. We’ve seen redesigns that looked better subjectively but decreased conversion rates by 15% to 20% because they introduced friction the design team didn’t anticipate. Split testing key templates before and during a redesign ensures you’re not trading a known performance level for an unknown one.

How A/B Testing Works

Running a valid A/B test requires more discipline than most teams expect. The difference between a test that produces actionable insights and one that produces noise comes down to five things: prioritization, hypothesis, sample size, duration, and measurement. A typical test follows this sequence:

  1. Pick a high-impact target. Choose a high-traffic page or high-value conversion step where a measurable improvement would affect revenue.
  2. Write a hypothesis. State the change, the predicted outcome, and why you expect it to work.
  3. Calculate the sample size. Determine how many visitors each variant needs before you launch.
  4. Build and QA the variant. Confirm tracking fires correctly and both versions render properly on every device.
  5. Split traffic randomly and run both versions concurrently. Leave the test untouched until it reaches its planned sample and duration.
  6. Analyze, document, and apply. Compare results on the primary metric, record what you learned, and roll out the winner or queue the next test.

Start with a hypothesis, not a hunch. A valid hypothesis connects a specific change to a predicted outcome with a rationale. “Changing the CTA button from ‘Submit’ to ‘Get My Free Quote’ will increase form completions because it communicates value rather than effort.” That structure forces you to think about why the change should work, not just what to change. Tests without hypotheses generate data but not understanding, which means you can’t apply the insight to other pages or channels.

Calculate your required sample size before launching. The most common A/B testing mistake is ending a test too early because one variant “looks” better. Statistical significance requires a minimum number of observations, and that number depends on your baseline conversion rate, the minimum detectable effect you care about, and your acceptable error rate (most teams use a 95% confidence threshold). Sample size calculators from testing platforms like Optimizely and VWO help determine what “enough data” actually means for your specific situation. Running a test for three days on a page that gets 50 visits per day doesn’t produce a valid result, regardless of how different the conversion rates look.

Run for full business cycles. Test duration matters independently of sample size. Even if you reach your target sample quickly, run every test for at least one full week, and ideally two, because conversion behavior varies by day of week, time of month, and even pay cycles. Apparent early winners frequently reverse once a full cycle of data comes in.

Control for external variables. Run both variants simultaneously, never sequentially. A test that runs version A in week one and version B in week two isn’t an A/B test. It’s a before-and-after comparison contaminated by every external factor that changed between those two weeks: seasonality, ad spend fluctuations, competitive dynamics, even day-of-week effects. True A/B testing requires concurrent, random assignment.

Measure the right thing. Your primary metric should connect directly to a business outcome. Click-through rate is a useful diagnostic metric, but it’s not the end goal. A headline that gets more clicks but attracts less qualified traffic might actually decrease revenue. Define your primary metric (form completions, purchases, qualified lead submissions) before launching, and resist the temptation to switch metrics mid-test because a different number looks more favorable.

Common pitfalls include testing too many things at once (which makes it impossible to attribute the result to a specific change), editing a variant after the test starts (which invalidates the data), stopping tests during periods of unusual traffic, and treating inconclusive results as failures. A well-run test that shows no significant difference between variants is still valuable: it tells you that the variable you tested doesn’t matter as much as you thought, freeing you to focus testing resources on higher-impact elements.

External Resources

Frequently Asked Questions

What is A/B testing in simple terms?

A/B testing is a method of comparing two versions of something to see which one performs better. You show version A to one group of people and version B to another, then measure which version gets more of the result you want, whether that’s clicks, form submissions, purchases, or any other goal. It removes guesswork from marketing decisions by letting real user behavior determine what works.

Is split testing the same as A/B testing?

Yes. Split testing and A/B testing are two names for the same method: dividing an audience at random between a control and a variant and measuring which performs better. The only nuance is split URL testing, where each variant lives on its own URL and visitors are redirected between them. That approach suits full page redesigns, and it should use canonical tags and temporary 302 redirects to avoid SEO issues.

What’s the difference between A/B testing and multivariate testing?

A/B testing compares two complete versions of a page or tests a single variable change. Multivariate testing evaluates multiple variables and all of their combinations simultaneously, which reveals how elements interact. The trade-off is traffic: multivariate tests need significantly more visitors because traffic is divided among many more combinations. For most businesses, A/B testing is the practical starting point.

How do I run an A/B test?

Start by identifying a high-traffic page or element where a measurable improvement would impact revenue. Form a hypothesis about what change you expect to improve performance and why. Use a testing platform (Optimizely, VWO, AB Tasty, or similar) to create your variant and split traffic randomly between the control and the variant. Let the test run for at least one full week and until it reaches statistical significance, then analyze the results against your primary metric, document what you learned, and apply the insight.

How does A/B testing connect to website optimization?

A/B testing is the primary methodology behind effective website optimization. Rather than redesigning pages based on assumptions, optimization programs use A/B tests to validate changes before rolling them out permanently. This applies to everything from headline copy and form design to page layout and navigation structure. The testing process ensures that every change to your website is backed by data showing it actually improves performance for your audience.

Does A/B testing hurt SEO?

Not when it’s implemented correctly. Google explicitly supports testing as long as you don’t cloak (show Googlebot different content than users), use rel=canonical on variant URLs, use 302 rather than 301 redirects for split URL tests, and end tests once they conclude. Done properly, testing and SEO reinforce each other: SEO drives the traffic, and testing makes sure the pages it lands on convert.

Is A/B testing only for websites?

No. A/B testing applies to virtually any digital marketing channel. Email marketers test subject lines, preview text, send times, and content structure. Paid media teams test ad copy, creative, audience segments, and bidding strategies. Even offline marketing elements like direct mail can be A/B tested by splitting recipient lists. The principle is channel-agnostic: wherever you can control the variable and measure the outcome, you can run a valid test.

How much traffic do I need for A/B testing?

The required traffic depends on three factors: your current conversion rate, the minimum improvement you want to detect, and your tolerance for statistical error. As a rough benchmark, most tests on pages converting at 2% to 5% need at least 1,000 to 5,000 visitors per variant to detect a meaningful difference. Low-traffic pages can still be tested if you run longer tests and focus on bold changes likely to produce larger effects. Below roughly 500 visitors per month, qualitative methods like heatmaps, session recordings, and user surveys usually identify improvements faster.

Related Resources

Related Glossary Terms

  • Conversion Rate: The percentage of visitors who complete a desired action. Conversion rate is the primary metric that A/B testing aims to improve.
  • Conversion Rate Optimization: The systematic process of improving conversion rates through research and experimentation. A/B testing is the core experimental method inside any CRO program.
  • Multivariate Testing: A method that tests multiple variables and their combinations at once. It extends A/B testing to more complex designs but needs far more traffic.
  • Landing Page: A standalone page designed for a specific campaign or conversion goal. Landing pages are among the highest-impact assets to A/B test.
  • Heatmap: A visual representation of where users click, scroll, and focus on a page. Heatmaps generate hypotheses that A/B tests then validate.
  • User Experience (UX): The overall experience a visitor has interacting with a website. A/B testing is one of the primary tools for measuring and improving UX.