A/B Testing in Email Marketing Campaigns

A/B Testing in Email: Key Takeaways from Klaviyo:BOS

Testing is a crucial part of getting more out of your email marketing campaigns.

It is an essential tool for all marketers to not only improve your emails but get to know your customers better — what kind of images, content, calls to action they engage most with.

Email A/B Testing: Beyond The Subject Line

These tips are suggested by Abby Siciliano, Senior Strategist, CRM & Email, from Elite SEM. Abby has worked in the industry for over 10 years with brands like Chico’s FAS, White House | Black Market, HSN, Everything But Water and many more.

She suggests that there are some clear reasons why you should be testing your emails.

A/B tests lead to clear direction for teams, eliminate decisions based on opinion, it’s a consistent path for growth and overall best in class customer experience.

But currently, according to a survey of marketers, only 31% regularly test and nearly 43% don’t know what to test.

There’s more to testing than just knowing what to test.

Here, we’ll look at four key types of tests, so that you can understand the range of options and pick what works best for you.

4 Types of Testing Structures

There are four key test types: A/B, Multivariate, Champion vs. Challenger, and Hold Out.

[1] A/B testing variations

A/B testing is when you test just two variations in your email. For example, changing a call to action in your emails.

You might test “Shop Now” vs “Shop the Sale” as the call to action in your email.

This type of test involves a 50/50 split, meaning half of your audience gets each variation. It’s the best type of test to use as a first testing approach for new ideas.

With an A/B test, you don’t want to review metrics too early or make decisions after only one test. It’s the perfect test for determining which of two options works best.

[2] Multivariate Testing

Multivariate Testing involves testing multiple variations to determine which combination of the variations is the most effective.

If you’re exhausted by the number of single A/B tests you are running, multivariate can be a great option to enhance your testing.

However, with this test you want to avoid testing multiple KPI success metrics and just focus on a single metric.

This can be a great way to optimize winning elements from prior tests.

[3] Champion vs. Challenger testing

This style of testing is a longer-term testing plan. Each winning variation is then run again as the control against a new challenger.
champion vs challenger testing flow chart
With this test type, it’s important to plan the test period to have breaks for implementation. Since it’s for your long-term strategy, you shouldn’t test during holiday or high seasons as it may skew results.

However, it’s great for ongoing testing of existing experience plus many new ideas.

[4] Hold Out testing

Hold Out testing basically involves “holding out” a control group from an email, meaning they are excluded.

Then you can measure the purchases of both groups to better evaluate the value of each group and understand the impact of your marketing.

This type of testing can very effective to better understand incrementally and ROI compared to other test types.

However, it should also be seen as a last resort type of testing and only used on the smallest group possible to avoid leaving money on the table.

Marketing Examples

A/B Click Test
blenders test
In this test from Blenders Eyewear, there are two very different visual styles with similar images and calls to action. Version A saw a +33% increase in clicks and a +48% lift in revenue.

A/B Cadence Test
cadence test
This test looked at email cadence. Variation B — with fewer emails — actually drove a +46% increase in orders.

Final Piece of Advice

There are many types of tests that can be run for a business. These can all have a great positive impact on your business. So get inspired and start testing!

Frameworks & Organization

It’s vital to have a strong framework and organizational plan when you start testing. We recommend having a single written record that is a “single source of truth” for your testing.

This record should include six key attributes:

  1. Status
  2. Test Idea
  3. Hypothesis
  4. Success Metric
  5. Result
  6. Follow Up

Having this record allows you to stay organized and have a picture of what has happened in previous tests you’ve run — and what you want to do next. Check out an example of the spreadsheet I use to track testing.

You can also think of tests in two buckets: Big Tests, also known as strategy testing, and Little Tests, also known as optimization testing.

Big Tests are good for marketing gut checks.

With a Big Test, you’re basically creating two completely different emails. This can be useful to make sure your overall email marketing strategy is going to the right direction. That said, it’s not something you should do frequently.

Little Tests are good to really optimize one part of your email.

By testing just one thing at a time, you can really see the full impact of the test you ran.

In this test type, it’s crucial to make sure your success metric for the test matches the test you are running.

For example, if you’re testing subject lines, you want to be looking at open rates whereas if you are testing calls to action you want to look at clicks and conversions.

Setting Up Tests in Klaviyo

There are a few different ways to create tests using flows and campaigns in Klaviyo.

In flows there are two basic ways to set up tests. You can set up a test with a single email or a flow branch. Both can be configured to run all the types of tests you’ve been reading about here.

With a single email test, you can impact all aspects of the email, such as design, content, and calls to action. With a flow branch test, you can test the number of emails, time between sends and much more.

For example, you might want to test how many emails result in the best experience for your

Abandoned Cart flow. You might find that your conversion rates really hit home on email 4 or that conversely you see a spike in unsubscribes at that point. Rather than sending everyone down the 4 email path, you can test it first on a smaller subset to make sure you’re not leaving money on the table.
flow ab test
Find out more about setting up flow tests in Klaviyo.

In a campaign you can test with a variety of features. Much like a single flow email, you can test any of the contents of the email.

Additionally, one of the newer types of tests you can run is by time. In a time based test, you want to keep the contents of the email the same in order to make sure your results based on the single variable of what time the email sent. You might want to test what time you should be sending your newsletter.

For example, you might want to test if your audience is more likely to read your newsletter over breakfast or after lunch!
campaign time test
Find out more about setting up campaign tests in Klaviyo.

Final Thoughts

A/B testing is an iterative process — not every test will have a meaningful result, and when one does, often the results will surprise you! Be systematic and disciplined, be creative, and most of all, be open to getting (and acting on) some unexpected results.

Back to Blog Home
Get email marketing insights delivered straight to your inbox.