What is Email A/B Testing & How to Maximize Your Conversion Rate

a/b testing

Consumers are shifting how they shop and spend money after the onset of the coronavirus (COVID-19). Some of these shifts may be temporary, while others may be more permanent.

Regardless of whether you’ve seen an uptick or drop in traffic to your ecommerce store, continuously testing conversion points on your website is vital to take advantage of current and potential customers that find their way to your site.

What is A/B Testing?

A/B testing is a way for you to compare two (or more) versions of a webpage to understand which one performs better.

To run a successful test, you must first come up with a hypothesis and set a goal for what you want to achieve in the test, typically based on a conversion rate or action that you determine is valuable. 

For example, your goal may be to increase the rate at which people add items to their cart by five percent. You might have a hypothesis that the location of your “add to cart” button on your product pages doesn’t stand out compared to the rest of the page, so you decide to test a new color for the button.

With the hypothesis and goal in place, it’s now time to set up the test targeting. The most frequent way to test is to show half of your visitors the original version (the control) and half the variation, but you can split the test any way you want.

Generally, you never want to allocate more than half of your traffic to go to the variation, since it comes with the risk of negatively impacting your website performance.

If you allocate less than half to the variation, the test will be seen as less risky, but can also take longer to reach statistically significant results.

As a general rule, you should always run your tests until you reach a minimum of 90 percent statistical significance, and you should always set a minimum timeframe for your test.

For example,  you might want to run the test for a week minimum to ensure early results aren’t skewed.

You can also narrow down your targeting options such as whether or not you want to show the test to first-time visitors, people in a certain geographical location, and other segments.

There are various ways you can think about targeting, but you should consult the documentation of whichever tool you’re using to determine which options are available to you.

Once the test is set up, it should run until statistically significant results are achieved rather than for a predefined period of time.

It’s also important to note that websites with high traffic will typically see results much quicker than websites with less traffic. 

When you have the results you’re looking for and can make a conclusion either in favor of the control or the variation, you can effectively end the test. 

If the control wins, you shouldn’t make any changes based on the test and you can go back to the drawing board to come up with a new hypothesis that’ll hopefully help you reach your goal. If the variation wins, then you can implement the winning version into your live website. 

If you’re limited by time and resources, you can set your A/B testing tool to display the variation to 100 percent of traffic after the winning variation is determined.

This is an effective way to temporarily show the winning version to everybody to achieve the results of the new version, but it shouldn’t be used as a permanent solution.

A/B testing tools to use

There are a variety of A/B testing tools available to use, each with their benefits and drawbacks. Choosing a tool should be based on your intended use cases, ecommerce platform, goals, resources available, and traffic volume. 

Below are several tools that you can use to get started with A/B testing:

Google Optimize: Google Optimize is a free tool by Google that automatically integrates with your Google Analytics account for easy reporting. Google Optimize is best for businesses just starting out who want to run simple tests.

VWO: VWO is an effective tool for small and medium-sized businesses that need a bit more control over their tests that they’re unable to get with Google Optimize.

Optimizely: Optimizely is a powerful testing tool built for the needs of enterprise businesses that remains easy to use.

AB Tasty: AB Tasty includes a full set of testing and reporting tools as well as predefined dynamic widgets that can help you to nudge visitors to taking an action.

Ecommerce platform app store:

Various ecommerce platforms’ respective app stores have A/B testing plugins you can use. A simple search on the app store can find a wide variety of results, such as Neat AB Testing on Shopify.


Leveraging Klaviyo forms in your A/B tests

Klaviyo released form analytics, which allows you to measure how your embedded, popup, or flyout forms are performing at an aggregate or individual level. 

This allows you to view data such as how many views a form received, how many people closed it, how many people submitted it, and the submit rate (calculated by taking the total number of submits divided by the total number of views).

Using Klaviyo forms combined with your A/B testing tool of choice is a powerful way to gather valuable insights into how your users engage with your site.

For example, let’s say you have an embedded form on your site that allows users to sign up for your newsletter.

This form has been achieving a two percent conversion rate, but you feel you could get it higher by providing more context of what type of content each newsletter delivers. 

You could set up an A/B test in your testing tool and create a variation of the form inside of Klaviyo by cloning the existing form and making your edits. 

Then, all you would need to do is embed the cloned form code into the variation section of your testing tool and turn on the test.

All of the relevant data such as the views and submits will be automatically recorded in Klaviyo and you should ideally see an event split of views (assuming you’re doing a 50/50 split).


A/B testing ideas

The sky’s the limit when it comes to tests, but it’s important to only test one variable at a time. 

For example, if you are testing a call to action (CTA) and you change the color, the position, and the verbiage, it’s impossible to know which of the changes led to an increase (or decrease) in click rate. But if you test them all separately, you may notice that the verbiage and color both led to an increase in clicks, but the position actually led to a decrease.

Here are some ideas to help you get started in determining your own hypotheses and goals for your tests: 

  • CTA Verbiage: This is one of the easiest tests to run and involves just making changes to your wording. Maybe you want to create a more action-oriented CTA like “Buy Now” instead of “Add To Cart”for your audience. This type of test only changes the text of the button.
  • CTA Color: Color is a powerful factor in CTAs and, ideally, it should contrast with the color scheme of your website in order to stand out. This will draw the users’ eye to the CTA and hopefully entice them to click.
  • CTA position: The layout of a page, particularly where CTAs are placed, is very important. On product pages, for example, a CTA should typically be on the right side, above the fold.
  • CTA Shape: The shape of a CTA can make a subtle yet meaningful difference in conversion rates, such as squares versus circles or image-based CTAs.
  • Form length: The length of your forms has a significant impact on conversion rates.
  • With more and more people browsing on smartphones today, long forms, particularly with text entry requirements, can deter visitors from completing them.
  • Testing shorter forms may lead to an increase in conversion rate. Likewise, if you add fields to your forms, you should always test them to make sure they don’t decrease your conversion rate.
  • Images: A picture says 1,000 words and images on a website can be a powerful way to entice people to take action.
  • Consider testing images of your product, images of people using your product, lifestyle images, stock images, stylized images, etc.
  • This is the best way to find out what resonates most with your audience.
  • Form required fields: Similar to form length mentioned above, adding field requirements can impact the completion rate of your forms.
  • Specifically, requiring a phone number has been proven to impact conversion rates significantly more than the requirements of other types of form fields.
  • When asking for any type of information, including phone numbers, make it clear why you’re asking for the information (such as if you want to send shipping updates via text) and how you plan on using the information.
  • Keep in mind that forms on your site do not need to be static—you can use segment targeted forms to show different messaging and different questions to different audiences.
  • Social proof: Adding social proof to your website, such as reviews or endorsements, is a great way to build trust with your audience. You should test social proof to determine which quotes and reviews resonate most with people.
  • Checkout process: The best rule of the web, according to author Steve Krug, is don’t make your user think.” Easy, transparent, and straightforward checkout processes are the best way to ensure that your users actually complete their transactions.
  • Navigation: The navigation menu on your site is extremely important in making it easy for users to find what they’re looking for.
  • Consider testing more or fewer links in your navigation or testing the categories of your navigation to make it as user-friendly as possible.

Final thoughts

A/B testing is a powerful way to increase the value of your website and digital marketing efforts. Once you get a few tests under your belt, you’ll likely want to consider an “always-on” testing strategy to incrementally improve your website conversion rates.

When it comes to digital marketing for ecommerce, you typically have two levers you can pull to increase sales:

– You can drive more traffic to your website

– or you can improve the conversion rate of the existing traffic that’s coming to your website (or ideally do both at the same time). 

But building traffic can take time and can be expensive, particularly if you rely heavily on advertising.

A/B testing is a great, relatively low-cost way of maximizing the traffic that’s coming to your site.


Back to Blog Home
Get email marketing insights delivered straight to your inbox.
*By entering your email address and clicking Subscribe, you consent to receive marketing emails (such as newsletters, blog posts, webinars, event invitations and new product updates) from Klaviyo from time to time. You can unsubscribe at any time by clicking on the “Unsubscribe” link at the bottom of our emails. For more information on how we process your personal information and what rights you have in this respect, please see our Privacy Policy.
Own your data.
Own your growth.
© 2022 Klaviyo. All rights reserved. Klaviyo and the Klaviyo logo are trademarks or registered trademarks of Klaviyo, Inc. or its affiliates.
Terms and Privacy Manage Cookies