A/B testing a vital part of email marketing campaigns
The most vital part of running successful email marketing campaigns is doing A/B testing.
It is an essential tool for all marketers to not only improve your emails but get to know your customers better — what kind of images, content, calls to action they engage most with. There are some straight forward reasons why you should be testing your emails.
A/B tests lead to clear direction for teams, eliminate decisions based on opinion, create a consistent path for growth, and overall best in class customer experience.
However, there’s more to testing than just knowing what to test.
Here, we’ll look at four key types of tests, so that you can understand the range of options and pick what works best for you.
Top four types of testing structures for email campaigns
There are four key test types: A/B, Multivariate, Champion vs. Challenger, and Hold Out.
 Understanding A/B testing variations for email marketing
The basic type of A/B testing is when you test just two variations in your email. For example, changing a call to action in your emails.
You might test “Shop Now” vs “Shop the Sale” as the call to action in your email.
This type of test involves a 50/50 split, meaning half of your audience gets each variation. It’s the best type of test to use as a first testing approach for new ideas.
With an A/B test, you don’t want to review metrics too early or make decisions after only one test. It’s the perfect test for determining which of two options works best.
 Email campaigns that require Multivariate Testing
Multivariate Testing involves testing multiple variations to determine which combination of the variations is the most effective. If you’re exhausted by the number of single A/B tests you are running, multivariate can be a great option to enhance your testing.
However, with this test you want to avoid testing multiple KPI success metrics and just focus on a single metric.
This can be a great way to optimize winning elements from prior tests.
 Creating email campaigns and using Champion vs. Challenger testing
This style of testing is a longer-term testing plan. Each winning variation is then run again as the control against a new challenger.
With this test type, it’s important to plan the test period to have breaks for implementation. Since it’s for your long-term strategy, you shouldn’t test during holiday or high seasons as it may skew results.
However, it’s great for ongoing testing of existing experience plus many new ideas.
 Sophisticated email Hold Out testing
Hold Out testing basically involves “holding out” a control group from an email, meaning they are excluded.
Then you can measure the purchases of both groups to better evaluate the value of each group and understand the impact of your marketing.
This type of testing can very effective to better understand incrementally and ROI compared to other test types.
However, it should also be seen as a last resort type of testing and only used on the smallest group possible to avoid leaving money on the table.
A/B testing email marketing examples
A/B Click Test
In this test from Blenders Eyewear, there are two very different visual styles with similar images and calls to action. Version A saw a +33% increase in clicks and a +48% lift in revenue.
A/B Cadence Test
This test looked at email cadence. Variation B — with fewer emails — actually drove a +46% increase in orders.
Brand Tip: There are many types of tests that can be run for a business. These can all have a great positive impact on your business. So get inspired and start testing!
Creating a testing frameworks & organizational flow for email campaigns
It’s vital to have a strong framework and organizational plan when you start testing. We recommend having a single written record that is a “single source of truth” for your testing.
This record should include six key attributes:
- Test Idea
- Success Metric
- Follow Up
Having this record allows you to stay organized and have a picture of what has happened in previous tests you’ve run — and what you want to do next. Check out an example of the spreadsheet I use to track testing.
You can also think of tests in two buckets: Big Tests, also known as strategy testing, and Little Tests, also known as optimization testing.
Big Tests are good for marketing gut checks.
With a Big Test, you’re basically creating two completely different emails. This can be useful to make sure your overall email marketing strategy is going to the right direction. That said, it’s not something you should do frequently.
Little Tests are good to really optimize one part of your email.
By testing just one thing at a time, you can really see the full impact of the test you ran.
In this test type, it’s crucial to make sure your success metric for the test matches the test you are running.
For example, if you’re testing subject lines, you want to be looking at open rates whereas if you are testing calls to action you want to look at clicks and conversions.
Setting Up Tests in an email marketing platform like Klaviyo
There are a few different ways to create tests using flows and campaigns in Klaviyo.
Setting up email flows
In flows there are two basic ways to set up tests. You can set up a test with a single email or a flow branch. Both can be configured to run all the types of tests you’ve been reading about here.
With a single email test, you can impact all aspects of the email, such as design, content, and calls to action. With a flow branch test, you can test the number of emails, time between sends and much more.
For example, you might want to test how many emails result in the best experience for your abandoned cart flow. You might find that your conversion rates really hit home on email 4 or that conversely you see a spike in unsubscribes at that point. Rather than sending everyone down the 4 email path, you can test it first on a smaller subset to make sure you’re not leaving money on the table.
Creating email Campaigns for testing
In a campaign you can test with a variety of features. Much like a single flow email, you can test any of the contents of the email.
Additionally, one of the newer types of tests you can run is by time. In a time based test, you want to keep the contents of the email the same in order to make sure your results based on the single variable of what time the email sent. You might want to test what time you should be sending your newsletter.
For example, you might want to test if your audience is more likely to read your newsletter over breakfast or after lunch!
A/B testing is an iterative process — not every test will have a meaningful result, and when one does, often the results will surprise you!
Be systematic and disciplined, be creative, and most of all, be open to getting (and acting on) some unexpected results.Back to Blog Home