A/B Testing in Email: Key Takeaways from Klaviyo:BOS
Did you miss out on Klaviyo:BOS? Or were you there…but couldn’t clone yourself to attend every session? Don’t worry — we got your back. This post is part of the series covering your key takeaways from the conference talks.
Testing is a crucial part of getting more out of your email. It is an essential tool for all marketers to not only improve your emails but get to know your customers better — what kind of images, content, calls to action they engage most with.
At Klaviyo:BOS, our 2-day conference, we included two sessions to help build out your A/B testing strategy: Email A/B Testing: Beyond The Subject Line and Testing & Optimization: A/B Testing in Klaviyo.
In this post, we’ll walk through each presentation focusing on actionable insights. Get ready to A/B test like a pro!
Email A/B Testing: Beyond The Subject Line
Our first talk on A/B testing was brought to us by Abby Siciliano, Senior Strategist, CRM & Email, from Elite SEM. Abby has worked in the industry for over 10 years with brands like Chico’s FAS, White House | Black Market, HSN, Everything But Water and many more. In her current role, she leads strategy for some of the world’s fastest growing and largest brands.
According to Abby, there are some clear reasons why you should be testing your emails. A/B tests lead to clear direction for teams, eliminate decisions based on opinion, it’s a consistent path for growth and overall best in class customer experience.
But currently, according to a survey of marketers, only 31% regularly test and nearly 43% don’t know what to test.
But there’s more to testing than just knowing what to test. Here, we’ll look at four key types of tests that Abby talked about, so that you can understand the range of options and pick what works best for you.
4 Types of Testing Structures
There are four key test types: A/B, Multivariate, Champion vs. Challenger, and Hold Out.
 A/B Testing
A/B testing is when you test just two variations in your email. For example, changing a call to action in your emails. You might test “Shop Now” vs “Shop the Sale” as the call to action in your email.
This type of test involves a 50/50 split, meaning half of your audience gets each variation. It’s the best type of test to use as a first testing approach for new ideas. With an A/B test, you don’t want to review metrics too early or make decisions after only one test. It’s the perfect test for determining which of two options works best.
 Multivariate Testing
Multivariate Testing involves testing multiple variations to determine which combination of the variations is the most effective.
If you’re exhausted by the number of single A/B tests you are running, multivariate can be a great option to enhance your testing. However, with this test you want to avoid testing multiple KPI success metrics and just focus on a single metric. This can be a great way to optimize winning elements from prior tests.
 Champion vs. Challenger
This style of testing is a longer-term testing plan. Each winning variation is then run again as the control against a new challenger.
With this test type, it’s important to plan the test period to have breaks for implementation. Since it’s for your long-term strategy, you shouldn’t test during holiday or high seasons as it may skew results. However, it’s great for ongoing testing of existing experience plus many new ideas.
 Hold Out
Hold Out testing basically involves “holding out” a control group from an email, meaning they are excluded. Then you can measure the purchases of both groups to better evaluate the value of each group and understand the impact of your marketing.
This type of testing can very effective to better understand incrementally and ROI compared to other test types. However, it should also be seen as a last resort type of testing and only used on the smallest group possible to avoid leaving money on the table.
Just A Few Examples
Abby walked through a number of real customer examples of tests and their results. You can see all the examples here in her slides from the talk. We’ll go through just a couple in this post.
A/B Click Test
In this test from Blenders Eyewear, there are two very different visual styles with similar images and calls to action. Version A saw a +33% increase in clicks and a +48% lift in revenue.
A/B Cadence Test
This test looked at email cadence. Variation B — with fewer emails — actually drove a +46% increase in orders.
Final Piece of Advice
There are many types of tests that can be run for a business. These can all have a great positive impact on your business. So get inspired and start testing!
Testing & Optimization: A/B Testing in Klaviyo
I co-presented the next talk: a practical session on how to get set up for success with A/B testing. In my experience in marketing operations at Klaviyo and other companies, I’ve created and run numerous tests. I was joined by my colleague Chris Rignoli, a product expert here at Klaviyo, for the talk.
Our talk echoed a number of Abby’s key points. It’s valuable to test continually and figure out the best types of test to run for your business. It is also important have clear plans when it comes to testing so that you can make the most of your results.
Frameworks & Organization
It’s vital to have a strong framework and organizational plan when you start testing. We recommend having a single written record that is a “single source of truth” for your testing.
This record should include six key attributes:
- Test Idea
- Success Metric
- Follow Up
Having this record allows you to stay organized and have a picture of what has happened in previous tests you’ve run — and what you want to do next. Check out an example of the spreadsheet I use to track testing.
As you now know, Abby went over four types of tests: A/B, multivariate, champion vs. challenger, and hold out. You can also think of tests in two buckets: Big Tests, also known as strategy testing, and Little Tests, also known as optimization testing.
Big Tests are good for marketing gut checks. With a Big Test, you’re basically creating two completely different emails. This can be useful to make sure your overall email marketing strategy is going to the right direction. That said, it’s not something you should do frequently.
Little Tests are good to really optimize one part of your email. By testing just one thing at a time, you can really see the full impact of the test you ran.
In this test type, it’s crucial to make sure your success metric for the test matches the test you are running. For example, if you’re testing subject lines, you want to be looking at open rates whereas if you are testing calls to action you want to look at clicks and conversions.
Setting Up Tests in Klaviyo
There are a few different ways to create tests using flows and campaigns in Klaviyo.
In flows there are two basic ways to set up tests. You can set up a test with a single email or a flow branch. Both can be configured to run all the types of tests you’ve been reading about here.
With a single email test, you can impact all aspects of the email, such as design, content, and calls to action. With a flow branch test, you can test the number of emails, time between sends and much more.
For example, you might want to test how many emails result in the best experience for your Abandoned Cart flow. You might find that your conversion rates really hit home on email 4 or that conversely you see a spike in unsubscribes at that point. Rather than sending everyone down the 4 email path, you can test it first on a smaller subset to make sure you’re not leaving money on the table.
Find out more about setting up flow tests in Klaviyo.
In a campaign you can test with a variety of features. Much like a single flow email, you can test any of the contents of the email.
Additionally, one of the newer types of tests you can run is by time. In a time based test, you want to keep the contents of the email the same in order to make sure your results based on the single variable of what time the email sent. You might want to test what time you should be sending your newsletter. For example, you might want to test if your audience is more likely to read your newsletter over breakfast or after lunch!
Find out more about setting up campaign tests in Klaviyo.
A/B testing is an iterative process — not every test will have a meaningful result, and when one does, often the results will surprise you! Be systematic and disciplined, be creative, and most of all, be open to getting (and acting on) some unexpected results.Back to Blog Home