This is the third post in a series about Klaviyo’s new Smart Send Time algorithm. Catch up on part 1 and part 2 to learn more about how this new feature answers the age-old question, “When’s the best time to send my emails?”
When is the best time to send your emails? It’s a question that’s plagued marketers ever since they started using email to communicate with customers.
Thanks to advancements in technology, marketers now have enormous amounts of data they can use to figure out the answer. But send time optimization has typically been a time-consuming, manual process or a sub-optimally automated process at best.
Klaviyo recently solved this problem with the release of Smart Send Time, an automated feature that uses our custom-developed smart optimization technique to learn the behaviors of your email recipients and tell you exactly when to send future messages to maximize your open rates. Our research showed A/B tests like those set up in Smart Send time were the only consistent and accurate predictor of optimal send time.
Here’s a look at how we developed the model behind Smart Send Time.
How Klaviyo developed a custom send time optimization algorithm
Most email service providers (ESPs) calculate optimal send time with black box algorithms, which don’t tell you how or why they’re making the decisions they make. They claim they provide the ability to predict optimal send time based on historical data, but their methods do not hold up in A/B tests. To develop Smart Send Time, Klaviyo used a different method.
Thirty-five brands worked alongside our product development and data science teams to during the research and development process. We iterated through several different models and techniques before finding the strategy that powers Smart Send Time.
And the results show this approach worked.
Smart Send Time provides a 10 percent, on average, lift in open rates, which is four times the lift we saw in those rates by using industry standard personalization techniques.
Trial and error: How we developed our model
To find a company’s best send time, we started by looking at its historical data to predict the best send time for its entire contact list.
We started with a simple regression model and we also used more complex machine learning approaches to calculate a company’s best send time based on the previous opening behaviors of their email recipients.
While our models worked really well with historical data, they failed to deliver each time we tested them with customers.
The same thing happened with personalized send time models.
We built personalized best send time models based on the hours that recipients had previously been most active with the brand’s emails. We used simple classification and complex clustering approaches to assign recipients their best send time.
While our aggregate experiments showed a 2.5 percent lift, we found a lot of strange results when we examined what was going on at an individual cluster level.
|Anonymized company’s industry||No optimization: Open rate||Personalized send time: Open rate||Lift from using personalization|
|Fashion + Apparel||18.0%||18.3%||+ 1.7%|
|Animal + Pet Care||31.6%||31.8%||+ 0.6%|
|Fashion + Apparel||27.2%||28.0%||+ 2.9%|
|Beauty + Cosmetics||17.0%||18.0%||+ 5.9%|
|Adult Products||16.5%||17.2%||+ 4.2%|
|Fashion + Apparel||20.1%||19.3%||- 4.0%|
|Median Lift: + 2.5%|
|Mean Lift + 1.9%|
|Standard Deviation = 3.4%|
In one experiment with an animal and pet care company, we A/B tested the recipient’s personalized send time against a 4:00 p.m. control time. We saw the following results for the clusters of morning and evening email openers:
The cluster of morning openers is on the left part of the image above. Morning openers are people whose past behavior shows they’re more active with email in the morning.
In the experiment, we sent them an A/B test of their 7:00 a.m. personalized time versus a 4:00 p.m. control time. Which time did they prefer? 4:00 pm. This means that the morning openers did not respond better at their personalized time. Their open rates were lower, meaning personalization was providing a negative lift.
On the right, you can see the cluster of evening openers. Based on their past behavior, they should be more active in the evening. But in the A/B test, both 10:00 p.m. and 4:00 p.m. performed exactly the same.
Why weren’t these recipients responding better at their personalized send times?
Our initial models failed because the dataset based on historical open rate data contained a lot of biases.
Open times cluster around the times that emails were previously sent, not the times they’re opened, so traditional models are biased towards predicting send times that are close to previous send times.
Brands also tend to send emails at certain times of the day, usually mornings, so there was almost no data on how afternoon and evenings send times performed in the dataset.
Additionally, people typically only open their emails once, so the signal we get when someone opens an email only partially represents the full set of times they are active with email.
At this point, we knew we had to approach things differently to develop Smart Send Time.
Historical data was biased and our models didn’t hold up in experiments. But, data science isn’t just about building really good models on the data we have. It’s also a field where we can design how we collect our data.
We knew the data we had was biased. So we thought, what if we could design a way to collect unbiased data? What if we changed the problem around? Instead of trying to use past data to make predictions, what if we could collect new data that wasn’t biased and let us make good predictions of optimal send time?
What is the Klaviyo Smart Send Time algorithm?
Klaviyo’s Smart Send Time algorithm automatically sets up and executes the A/B tests that are required to determine an optimal send time. A/B tests are the only consistently accurate predictors of how recipients react to send time.
Smart Send Time first explores how your customers respond to emails at different send times by sending them over a 24-hour time period. It then narrows that period down to a smaller, more focused window of time and sends your emails at the best time while continuously collecting more data to determine if the send time is optimal or if it needs to be adjusted.
The A/B tests allow us to directly compare different send times and they eliminate biases in the data because they test different times throughout the entire day in a true head-to-head comparison.
During a private beta test, we tested Smart Send Time with 15 companies. These brands saw a 10 percent average increase in open rates by sending their emails at their optimal send time.
|Anonymized Company’s Industry||No Optimization: Open rate||Smart Send Time: Open rate||Improvement|
|Home + Garden||19.3%||21.7%||12.4%|
|Beauty + Cosmetics||13.2%||15.7%||18.9%|
|Health + Fitness||20.3%||20.6%||1.5%|
|Beauty + Cosmetics||14.2%||17.3%||21.8%|
|Jewelry + Accessories||8.5%||9.1%||7.1%|
|Fashion + Apparel||22.7%||25.5%||12.3%|
|Home + Garden||18.3%||18.7%||2.2%|
|Outdoor + Wilderness||6.7%||7.1%||6.0%|
|Beauty + Cosmetics||15.2%||17.4%||14.5%|
|Health + Fitness||6.5%||6.7%||3.1%|
|Jewelry + Accessories||18.0%||19.7%||9.4%|
|Jewelry + Accessories||19.9%||21.9%||10.0%|
|Health + Fitness||18.4%||20.6%||12.0%|
|Fashion + Apparel||3.8%||4.0%||5.2%|
|Median Lift: +10%|
|Mean Lift +10%|
|Standard Deviation = 6%|
How data science can help you own your brand’s growth
Klaviyo’s mission is to help brands grow. To do that, we want you to have the power to send emails to your recipients at the best possible time so you can maximize your open rates. We want you to have the tools you need to automatically test and act on your results. And we don’t want any of your data or insights to be hidden inside an algorithm.
Smart Send Time is fully transparent about the different times your emails will be sent during the tests. That’s your data, you own it, so you should be able to see it.
How do we know your Smart Send Time is the best time to send emails to your recipients? Because you tested it with the Smart Send Time algorithm! You did the experiment. You own the results. You found the optimal send time for your audience.
Learn more about how you can start using Smart Send Time today.
Back to Blog Home