Subject Line A/B Tests: How Emojis and Personalization Affect Your Open Rates
Imagine you’ve written an email full of great content—there’s engaging copy, stunning images, and a clear call to action. You’ve already decided on your target audience and the only thing left to create is a killer subject line that’ll drive open rates.
That’s easier said than done.
Should you call the recipient by their first name in the subject line? Should you include an emoji? 🤔
Some marketing experts argue that using emojis in your email subject lines can help you create a stronger brand voice. Others say that personalizing your email subject lines can help you connect with your audience.
Your email open rates may benefit from the use of emojis or personalization in your subject lines, but there’s no guarantee that these strategies will help.
In fact, they might actually hurt your email open rates. So how do you decide what the best option is for your brand?
To help you answer this question, I analyzed email campaign A/B tests from thousands of Klaviyo customers over the last four years.
The first set of A/B tests I considered were ones that tested email personalization in the subject line. In these tests, one email version was personalized and one was not. For example, one email subject line variation might read, “Our summer collection is here!” and the other would read, “Hey Julie, our summer collection is here!”
The second set of A/B tests I considered were email campaigns that tested the use of emojis in the subject line. For these tests, the subject line of one email variation might read, “Summer is here!” and the other would read, “Summer is here! 😎”
My findings in both tests were very interesting. Read on to discover:
- How personalization in email subject lines affects open rates
- The impact of using emojis in email subject lines
The first thing I noticed when I analyzed the dataset containing the A/B tests on subject line personalization was that the personalization in almost every test—approximately 90 percent of them—was the use of the customer’s first name.
This makes sense because first and last names are probably the most readily available data you have about your customers, second to their email.
When I analyzed the effect of using the recipient’s first name in the email subject line, I began by examining all the data at once.
I wanted to understand if and how often the personalized version of the email had a higher open rate than the non-personalized one.
|A/B test results||Non-personalized||Personalized|
|Number of emails that had the higher open rate||4,751 (45%)||5,925 (55%)|
In this scenario, it seems that email personalization can help but it’s not necessarily guaranteed to do so.
Since I didn’t learn very much from examining the data in this way, I changed my approach.
What if instead of considering all the email A/B tests at once, I separately counted the percentage of email personalization tests run by each company that ended with the personalized version as the winner.
Potentially the data could show that for half of the companies, email personalization always wins, and for the other half it always loses. But I can’t know that by analyzing all the A/B tests at once.
A coin toss experiment can help illustrate this point.
Coin tosses and email personalization
How does a coin toss experiment help explain the results of an email personalization A/B test?
That’s a great question. Here’s how the two experiments relate:
Say you flip a coin 10 times in a row and each time the coin lands head-side up. Would you say that’s a fair coin? There’s a one in a thousand chance that you’re just extremely lucky, but it’s more likely that the coin is biased.
When I say that the coin is biased, I mean that it has a preference for either heads or tails and that the toss pattern of the coin differs from the expected toss pattern of a fair coin.
So if a coin has a bias towards heads, I would expect to see more than 50 heads in 100 coin flips.
If you toss a coin ten times in a row, it’s reasonable to expect that you have a higher chance of five out of the ten tosses landing on heads than landing on heads ten times (or zero times) out of the ten tosses—as modeled by the probability distribution below.
If you add up the percentages of probability for all the green bars in the middle section of the graph the value would equal about 95 percent.
This means about 95 percent of the time a fair coin will land on heads between three and seven times for every ten tosses.
If you land on heads less than three times or more than seven times (i.e., your data falls to the left or right of the middle section) then you have some evidence that the coin is biased—in the words of statisticians, there is evidence of bias at the “95 percent confidence level.”
Each business that I analyzed has a different number of A/B tests (i.e., coin tosses), and a different number of times the email variation that was personalized won (i.e., the coin landing head-side up).
If the results of a company’s email personalization A/B test falls in the middle section of the above graph, their email open rates are essentially unaffected by personalization.
If the results fall on the left or right side sections, then the company’s emails have a significant decrease or increase in open rate, respectively, as a consequence of including personalization in their subject line.
In order to conduct this experiment and generate the desired 95 percent confidence level, you would need to toss a coin a minimum of six times.
So when I applied the coin toss experiment to the analysis of how email personalization affected open rate performance, I excluded all companies with fewer than five tests.
The impact of email personalization on open rates
If email personalization is as random as a fair coin toss, you would expect 95 percent of companies to see no impact on their open rates and approximately 2.5 percent to see either an increase or decrease to their open rates.
Here are the results from Klaviyo customers:
|Impact of adding personalization||Number of companies||Percentage|
|Significantly increased open rates||50||11.3%|
|Significantly decreased open rates||21||4.7%|
Based on the data, you can see that there are positive effects from the use of email personalization on open rates—11.3 percent is higher than 2.5 percent. This means that, at least for some companies, sending emails with personalized subject lines consistently results in higher open rates.
As expected, the distribution of results is more spread out than the random distribution seen in the coin toss experiment.
If customers were equally indifferent to personalization across all companies, you would expect the distribution of companies in each band to be 2.5 percent, 95 percent, and 2.5 percent, just like in the coin toss example.
But customers of different companies behave differently. Some audiences are very receptive to a personalized email subject line, significantly increasing the shift towards positive results.
In other instances, the audience’s preference (the bias) leans away from personalization, increasing the number of companies that see a negative impact on email rates.
But my analysis didn’t stop there. I was curious if the preference of the audience over time impacted the results.
I grouped all the tests by the month they were sent and then computed the percent of tests in which the personalized version of the email had resulted in higher open rates.
I also computed the 95 percent confidence interval for the same time period to use as a comparison (i.e., the middle section of the coin toss plot). The more A/B tests in a given month, the narrower the confidence interval gets.
Based on my findings, it would appear that over time the preference of individuals has not changed—your email recipients consistently enjoy the use of their first name in your email subject line (or at least were indifferent to it).
Personalizing your email subject line may be more likely to improve your open rates than harm them, but how does the use of emojis in an email subject line affect open rates?
I ran the exact same analysis as I did in the previous section on the use of emojis in email subject lines and these were the results:
|A/B test results||No emojis||Emojis|
|Number of emails that had the higher open rate||20,535 (55%)||16,926 (45%)|
Based on the data, it appears that the email variations without emojis in the subject lines performed better.
When I assessed the impact of adding an emoji to the subject line across different companies, the results varied from the email personalization analysis.
|Impact of adding emojis||Number of companies||Percentage|
|Significantly increased open rates||25||1.9%|
|Significantly decreased open rates||121||9.3%|
The data suggests that email recipients don’t favor the use of emojis in email subject lines. But again, the distribution of data is more spread out than the random distribution the coin tosses would generate.
Did the email recipients’ preference of emoji use change over time?
The data would suggest that audiences’ preference over time has not changed.
It seems that, contrary to popular belief, using emojis will not magically improve your open rates.
But it’s worth noting that in the dataset I analyzed, I found companies that were representative of both extreme ends of the spectrum.
One company performed 63 A/B tests using emojis in the subject line, and in only 13 of those tests did the subject line containing emojis win, or have a higher open rate. That’s a 20 percent win rate and an indication that the use of emojis in their subject lines doesn’t resonate with their audience.
Another company in my dataset performed 68 A/B tests and in 49 of them, the email variation with the emoji in the subject line won. That’s a 70 percent win rate.
On average though, the majority of A/B test results I analyzed mirrored those of a fair coin toss. One company conducted 220 emoji A/B tests. The use of emojis in the subject line won in 111 tests, and lost in 109 tests, which equals about a 50 percent win rate.
A/B test email subject lines to determine if personalization and emojis work for you
Based on the results of analyzing thousands of email campaigns, I found that personalizing email subject lines can improve open rates and the use of emojis in email subject lines can slightly decrease open rates.
Does that mean you should always use personalization in your email subject lines and never again add an emoji? Not necessarily.
What it means is that there’s no best practice here to live and die by.
Emojis aren’t a guaranteed solution to improve open rates and while personalizing email subject lines may help your email performance, testing both strategies and seeing how receptive your audience is to each one (i.e., learn which way your coin is biased) is the best way to know what works for you.
Also, there is power in knowing that something doesn’t make a difference. If you learn that your audience is indifferent to emojis in your email subject lines, have fun with them and use them as you see fit!
Want to learn more about A/B testing? Check out these signup form A/B tests that DTC brands swear by!
Start A/B testing your email subject lines today.