How to master A/B testing: nail the basics, then put optimization on autopilot

Profile photo of author Tiffany Regaudie
Email marketing
February 22, 2024
In lavender text on a lemon background, copy on the right side of the image reads, "how to master A/B testing." In smaller black font underneath that, copy reads, "Nail the basics, then put optimization on autopilot." On the left side of the image is an outline of the Klaviyo flag in lavender, and overlapping it on the left is a dark mode screenshot of an A/B test for an SMS campaign in the back end of Klaviyo.

For years, marketers have used A/B testing to optimize marketing assets.

In 2012, Bing performed a small A/B test on the way their search engine displayed ad headlines. It led to a 12% increase in revenue—more than $100M in the US alone.

But that was over a decade ago. Since then, getting attention online has become a lot more competitive. Noah Kagan, founder of web app deals website Appsumo, recently revealed that only 1 in 8 of their A/B tests produces significant results—meaning brands now have to perform these tests 1) at scale and 2) beyond email to keep iterating at a competitive rate.

This is where advanced A/B testing can help.

A/B testing (or split testing) is a research methodology that involves testing variations of a marketing email, SMS message, landing page, or website form to determine which one yields higher open rates, click rates, conversion rates, and more.

Now, new A/B testing technologies, including AI, empower brands to strip away some of the manual work that used to come with A/B testing—and set steps on autopilot to start seeing results faster.

Here, we’ll get into some of the basics of A/B testing, but we’ll also talk about some of the ways your brand can use new methodologies to speed up those tests—and ultimately get more out of them.

What do you test in A/B testing?

Important note: When you’re performing an A/B test, never test more than one variable at a time. Doing so will yield an inconclusive result, because you won’t know which variable influenced performance.

Email A/B testing: what to test

Whether you’re A/B testing email campaigns or A/B testing email automations, here are a few elements to consider testing:

  • “From” name (sender’s name that appears in an email’s inbox preview): your organization’s name vs. a more casual name (such as the first name of one of your team members) vs. a specific department of your organization, etc.
  • Subject line: length, format, personalization elements, emojis, etc.
  • Media: visuals vs. text, different hero images, gifs vs. static images, videos vs. infographics, etc.
  • Email design and layout: single column, multi-column, image grids, F-pattern, zig-zag, different email templates, etc.
  • Email content: offer and discount types, different user-generated content, voice and tone, length, emojis, etc.
  • Calls to action (CTAs): number, placement, copy, color, buttons vs. hyperlinks, etc.
  • Send time: day of week, time of day, etc.

How the right tech can make this easier—and make you more money

With a platform like Klaviyo, you don’t have to manually switch to sending the most successful variation of your A/B test.

Instead, you set the success criteria (more on this later) and/or the test duration, after which Klaviyo declares the winner automatically. The rest of the recipients will automatically receive the winning email.

Image shows the A/B test success metric selection screen in the back end of Klaviyo. The options under “Winning Metric” are open rate and click rate, which is the selected and recommended option in this screenshot. Next is a section called “Automatic Winner Selection,” which specifies, “The test will end automatically if you have any of the following settings selected. You can always manually end the test at any time.” In this screenshot, the first option, “End test if click rate is determined to win with high statistical certainty,” is checked. “Set a date limit” is not checked.

SMS A/B testing: what to test

A/B testing SMS messages alongside email is a smart way to continually optimize your marketing campaigns. Like with email, you can start A/B testing your SMS messages for varying elements, including:

  • SMS content: copy, offer and discount types, voice and tone, length, etc.
  • CTA: placement, copy, etc.
  • Media: images, gifs, memes, emojis, etc.
  • Send time: day of week, time of day, etc.
Image shows a gif of the SMS A/B testing screen in Klaviyo. After typing in their SMS message content, the user clicks on “Create A/B test,” then selects which content they want to test from a dropdown menu.

How the right tech can make this easier—and make you more money

If you use the same platform for your email and SMS marketing, you can test how email influences your SMS campaigns (and vice versa) by testing email-SMS order within an automated series or multi-part campaign. You can’t perform a test like this with a point solution.

That said, never try to test SMS against email. The channels are too different to yield a conclusive test—remember, A/B testing is only valid when you test one element at a time.

What is SMS marketing? Your guide to the fastest-growing ecommerce marketing channel
Not sure SMS is right for your brand? There are other ways to start small before investing big.

Forms A/B testing: what to test

Sign-up forms are vital for building an email list—so why wouldn’t you test them with the same rigor as your marketing communications?

Here are a few common elements for testing your sign-up forms:

  • Type of form: embedded vs. pop-up vs. full-screen vs. fly-out
  • Audience behavior: first-time visitors vs. returning visitors, scroll depth, time lapse, exit intent, etc.
  • Content: design elements, copy, CTA button placement and copy, number of fields, etc.
  • Type of incentive: free gift vs. free shipping vs. percent off vs. dollar amount off vs. early access

How the right tech can make this easier—and make you more money

With Klaviyo, forms display optimization uses AI to put testing your sign-up forms on autopilot. It’s basically GPS for web forms—when you use this type of test, you don’t have to manually set parameters and choose a winner.

Instead, Klaviyo will run a series of AI-generated experiments to determine the most effective display time for conversion. Multiple versions of your form will display at various time delay, exit intent, and scroll percentage combinations.

The optimization process will continue running until AI finds the best display timing. After all AI-powered experiments are done, the algorithm chooses a winner—and that version is the one that will display on your website moving forward.

Image shows the forms display optimization screen in the back end of Klaviyo. On the left side of the screen is the test’s progress so far (in “Phase 1: Learning,” which is marked “in progress,” with date and time markers). The remaining phases are “Iteration phases” and “Test completed.” On the right side of the screen are statistics for “Submit rate lift” and “Win probability,” as well as “Traffic distribution,” divided by control traffic and test traffic.

Measuring A/B testing success: why statistical significance matters

Before running an A/B test, you need to decide how you’ll determine the winner. To that end, choose a performance metric based on the A/B test variable. For example:

  • Open rate: if you’re testing “from” name, subject line, or preview text
  • Click rate: if you’re testing content like email layout, different visuals, CTA appearance, or CTA copy
  • Placed order rate: if you’re testing content like social proof, send time, or CTA placement

But after you run your tests and collect the results, how do you interpret them? Assume you A/B test two email layouts. Version A earns a 15% click rate, whereas version B earns a 14% click rate. Does that mean version A is the winner?

A/B testing is a method of statistical inference where you arrive at a result via the behavior of a sample. That means you need to make sure the results are statistically significant.

How the right tech can make this easier—and make you more money

You don’t have to do this manually—you can lean on a platform like Klaviyo to automatically classify statistically significant results, so you can focus on the tests that teach you something.

Klaviyo categorizes A/B test results into 4 groups:

  • Statistically significant: A certain variation of your test is highly likely to win over the other option(s). You could reproduce the results and apply what you learned to your future sends.
  • Promising: One variation looks like it’s performing better than the other(s), but the evidence isn’t strong enough based on one test. Consider running another.
  • Not statistically significant: One variation beats the other(s) in the test, but only by only a slight amount. You may not be able to replicate the result in another test.
  • Inconclusive: There’s not enough information to determine whether or not something is statistically significant. In that case, you may want to expand your recipient pool or follow up with more tests.
Image shows a decision tree for determining whether an A/B test is statistically significant. To be statistically significant, the win probability must be 90% or more. To be promising, it must be 75% or more. If the test achieves neither probability and the test size has at least 1800 x the number of variations of recipients, it’s not statistically significant. If the percentage difference between the winning and next-best variation is 4% or less, and the win probability is 60% or less, it’s not statistically significant. Otherwise, the test is inconclusive.

7 best practices for A/B testing emails, texts, and forms

To get the most of your A/B testing program, you’ll want to establish a set of criteria to follow, just as any scientist would when they’re running an experiment.

Here are a few A/B testing best practices to keep in mind:

1. Develop a hypothesis

Before choosing which variables to test, create a hypothesis based on what you’re trying to achieve with your A/B testing.

You may develop a hypothesis like:

  • A subject line for my abandoned cart automation will drive more placed orders when it references the name of the product.
  • An email that looks like it’s coming from a person rather than our company name will earn more opens.
  • An email with a contest entry offer will perform just as well as a discount offer—and will cost the company less.
  • A text message will drive more clicks when it’s preceded by an email.
  • A sign-up form will drive higher list growth if it appears after the shopper has scrolled at least 75% of the page.

Your hypothesis directly informs the elements you choose to test.

2. Use a large sample

Imagine you A/B test a form. On the first day, all 10 people who see the form click through and make a purchase. But over the course of a week, the next 2K people exit out of the form immediately. If you ended the test after the first day, you would think your form was great, even though a larger audience showed otherwise.

A/B testing is a statistical experiment where you derive insight from the response of a group of people. The larger the sample size of the study, the more accurate the results. Testing across too small a group may result in statistically insignificant or inconclusive results.

Always wait until your results are statistically significant or until you have a good sample size of viewers before ending your test. Klaviyo recommends setting your test size proportional to your list size or average website traffic, such as 20% of your list for the first version of your email and 20% for the second.

Image shows the A/B test screen in the back end of Klaviyo where the user selects the test size. They have set Version A to appear to 20% of their list, Version B to appear to 20%, and 60% to receive the winning version. Next is a section called “Test duration,” which the user has set from a dropdown menu to 6 hours and 0 days.

3. Test low-effort elements first

When you’re first starting out, choose high-impact, low-effort elements that take less time to test. For instance:

  • Subject line or headline
  • CTA button copy
  • Emojis
Image shows two different versions of an email from Comrad, headlined, “Dog days of summer.” Both emails feature the same headline, body copy, and photos—the only difference is the CTA button, which reads “get yours” in the first version and “shop parachutes” in the second.
Source: Klaviyo email marketing roadmap

Once you’re done with the low-effort elements, you can test higher-effort ones, like:

  • Format or layout
  • Copy length and detail
  • Different types of visuals
Image shows an example of an A/B test in an email. The two emails are identical other than the photography: one features a close-up of a dog, and the other features the dog and their owner on a beach. Automation software has tagged the first email with a label that says “.21% placed order rate,” and the second with a label that says “.10 placed order rate.”

4. Prioritize the messages you send the most

Invest your A/B testing efforts into the marketing that yields the most profit. Start with perfecting the messages you frequently send, which may be your:

  • Abandoned cart emails and texts
  • Promotional emails and texts
  • Welcome emails and texts

You can also look at your historical list and segment sizes, and prioritize the marketing messages that are seen by the most amount of people. Focusing on these communications ensures you reach many recipients and get a better return on investment (ROI) on your A/B testing efforts.

5. Wait enough time before evaluating performance

SMS subscribers tend to read texts right away. But even if you’ve sent an email to a large group of people, they might not engage with it immediately. Most people need time to get around their email inbox, so make sure you wait long enough for your recipients to engage.

Klaviyo recommends setting a test duration based on the A/B test metric you’re evaluating—for example, if you’re using placed orders as your success metric for an email A/B test, your test duration will probably be longer than if you’re evaluating based on opens or clicks.

6. Test a single variable at a time

You may have many ideas for A/B testing your emails, texts, and forms. But it’s best to test one variable at a time. Testing more than one variable simultaneously makes it difficult to attribute results.

Let’s say you edit the color of the CTA and also the copy on a sign-up form, and you notice a spike in form submit rates. You won’t know if the increased engagement was the result of the change in color or the copy.

By testing one variable at a time, you can be sure you understand exactly what it is your audience is responding to.

7. Test no more than 4 variations at a time

In the same vein, the more variations you add, the bigger the audience you’ll need to make sure you get reliable results.

We recommend testing up to 4 variations of the same variable at a time—for example, the same CTA button in 4 different colors. If you’re concerned that your sample size isn’t large enough to accommodate that many variations, stick with 2-3.

Power smarter digital relationships.
Try Klaviyo
Tiffany Regaudie
Tiffany Regaudie
Tiffany is a writer and content consultant who specializes in marketing, health, and the attention economy. Before devoting herself to freelance writing full-time, she led content teams at various startups and nonprofits in Toronto, Canada.