A/B testing a vital part of email marketing campaigns

Ab testing email campaigns

The most vital part of running successful email marketing campaigns is doing A/B testing.

It is an essential tool for all marketers to not only improve your emails but get to know your customers better — what kind of images, content, calls to action they engage most with. There are some straight forward reasons why you should be testing your emails.

A/B tests lead to clear direction for teams, eliminate decisions based on opinion, create a consistent path for growth, and overall best in class customer experience.

For email marketing A/B testing refers to the sending variations of the same email campaign to different segments of your subscribers to understand which one gets better results.

However, there’s more to AB testing than just multivariate tests.

Website A/B testing that helps improve email campaigns

Here, we’ll look at four key types of tests, so that you can understand the range of options and pick what works best for you.

Top four types of testing structures for email campaigns

There are four key test types: A/B, Multivariate, Champion vs. Challenger, and Hold Out.

[1] Understanding A/B testing variations for email marketing

The basic type of A/B testing is when you test just two variations in your email.

For example, changing a call to action in your emails.

You might test “Shop Now” vs “Shop the Sale” as the call to action in your email.

This type of test involves a 50/50 split, meaning half of your audience gets each variation. It’s the best type of test to use as a first testing approach for new ideas.

With an A/B test, you don’t want to review metrics too early or make decisions after only one test. It’s the perfect test for determining which of two options works best.

[2] Email campaigns that require Multivariate Testing

Multivariate Testing involves testing multiple variations to determine which combination of the variations is the most effective. If you’re exhausted by the number of single A/B tests you are running, multivariate can be a great option to enhance your testing.

However, with this test you want to avoid testing multiple KPI success metrics and just focus on a single metric.

This can be a great way to optimize winning elements from prior tests.

[3] Creating email campaigns and using Champion vs. Challenger testing

This style of testing is a longer-term testing plan. Each winning variation is then run again as the control against a new challenger.
champion vs challenger testing flow chart
With this test type, it’s important to plan the test period to have breaks for implementation. Since it’s for your long-term strategy, you shouldn’t test during holiday or high seasons as it may skew results.

However, it’s great for ongoing testing of existing experience plus many new ideas.

[4] Sophisticated email Hold Out testing

Hold Out testing basically involves “holding out” a control group from an email, meaning they are excluded.

Then you can measure the purchases of both groups to better evaluate the value of each group and understand the impact of your marketing.

This type of testing can very effective to better understand incrementally and ROI compared to other test types.

However, it should also be seen as a last resort type of testing and only used on the smallest group possible to avoid leaving money on the table.

A/B testing email marketing examples

A/B Click Test
blenders test
In this test from Blenders Eyewear, there are two very different visual styles with similar images and calls to action. Version A saw a +33% increase in clicks and a +48% lift in revenue.

A/B Cadence Test
cadence test
This test looked at email cadence. Variation B — with fewer emails — actually drove a +46% increase in orders.

Brand Tip: There are many types of tests that can be run for a business. These can all have a great positive impact on your business. So get inspired and start testing!

Creating a testing frameworks & organizational flow for email campaigns

It’s vital to have a strong framework and organizational plan when you start testing. We recommend having a single written record that is a “single source of truth” for your testing.

This record should include six key attributes:

  1. Status
  2. Test Idea
  3. Hypothesis
  4. Success Metric
  5. Result
  6. Follow Up

Having this record allows you to stay organized and have a picture of what has happened in previous tests you’ve run — and what you want to do next. Check out an example of the spreadsheet I use to track testing.

You can also think of tests in two buckets: Big Tests, also known as strategy testing, and Little Tests, also known as optimization testing.

Big Tests are good for marketing gut checks.

With a Big Test, you’re basically creating two completely different emails. This can be useful to make sure your overall email marketing strategy is going to the right direction. That said, it’s not something you should do frequently.

Little Tests are good to really optimize one part of your email.

By testing just one thing at a time, you can really see the full impact of the test you ran.

In this test type, it’s crucial to make sure your success metric for the test matches the test you are running.

For example, if you’re testing subject lines, you want to be looking at open rates whereas if you are testing calls to action you want to look at clicks and conversions.

Setting Up Tests in an email marketing platform like Klaviyo

There are a few different ways to create tests using flows and campaigns in Klaviyo.

Setting up email flows
In flows there are two basic ways to set up tests. You can set up a test with a single email or a flow branch. Both can be configured to run all the types of tests you’ve been reading about here.

With a single email test, you can impact all aspects of the email, such as design, content, and calls to action. With a flow branch test, you can test the number of emails, time between sends and much more.

For example, you might want to test how many emails result in the best experience for your abandoned cart flow. You might find that your conversion rates really hit home on email 4 or that conversely you see a spike in unsubscribes at that point. Rather than sending everyone down the 4 email path, you can test it first on a smaller subset to make sure you’re not leaving money on the table.

Ecommerce marketing automations for email marketing


flow ab test

Creating email Campaigns for testing
In a campaign you can test with a variety of features. Much like a single flow email, you can test any of the contents of the email.

Additionally, one of the newer types of tests you can run is by time. In a time based test, you want to keep the contents of the email the same in order to make sure your results based on the single variable of what time the email sent. You might want to test what time you should be sending your newsletter.

For example, you might want to test if your audience is more likely to read your newsletter over breakfast or after lunch!

Email marketing campaigns that can be tested for increased click through rates


campaign time test

Final Thoughts

A/B testing is an iterative process — not every test will have a meaningful result, and when one does, often the results will surprise you!

Be systematic and disciplined, be creative, and most of all, be open to getting (and acting on) some unexpected results.

Email marketing and website sign-up form examples and testing that helps grow your subscriber list

Back to Blog Home
Get email marketing insights delivered straight to your inbox.
*By entering your email address and clicking Subscribe, you consent to receive marketing emails (such as newsletters, blog posts, webinars, event invitations and new product updates) from Klaviyo from time to time. You can unsubscribe at any time by clicking on the “Unsubscribe” link at the bottom of our emails. For more information on how we process your personal information and what rights you have in this respect, please see our Privacy Policy.
Own your data.
Own your growth.
© 2022 Klaviyo. All rights reserved. Klaviyo and the Klaviyo logo are trademarks or registered trademarks of Klaviyo, Inc. or its affiliates.
Terms and Privacy Manage Cookies