A Note from Klaviyo’s CEO
There are a lot of things I love about my job. The fact that I, and every member of the Klaviyo team, can help our customers deliver better results for their companies tops that list. I’m incredibly proud to play even a small part in the success of their business. And I know how important Black Friday and Cyber Monday are to every retailer, which is why we invested a lot of time preparing and testing our infrastructure to prepare for the start of the holiday marketing season.
But despite those preparations, we fell short. We delivered a poor experience on one of the most important days of the year. I’ve apologized and sent this note to all of our customers, but I’m also sharing it publicly because I want to make sure our community at large understands what happened, and what we’ve done to prevent it from happening again.
There were three types of issues Klaviyo ran into on Cyber Monday: email sending delays, app accessibility, and event processing delays.
Email sending delays
We were equipped to process 10x our average sending volume over the holiday weekend. On Monday morning around 9am, we noticed email sending was not progressing as quickly as expected, and we started adding additional capacity immediately. Campaigns sent between approximately 9am and noon ET on Monday, however, may have experienced sending delays of up to 5 hours. I would like to reassure you that all campaigns sent on Cyber Monday were delivered. I understand how time sensitive promotions are, particularly on Cyber Monday, and the delays we experienced were unacceptable.
Around 11am ET on Monday, the capacity we added to improve email sending led to a spike in hits to one of our databases, and we started experiencing some connectivity issues. That resulted in parts of the Klaviyo UI being inaccessible for up to one hour. This did not impact any scheduled campaigns or flows, and we were able to resolve it by adding capacity to the impacted database.
Event processing delays
The high volume of activity also led to a backup in analytics processing in the Klaviyo application. This led to issues across reporting, AB testing, and segment processing:
- On the reporting side, the fact that events were taking longer than usual to process meant that campaign and flow performance reporting was not real-time. Things like opens, clicks, and sales all appeared to be lower than they actually were. This has been resolved, and the results from yesterday’s campaigns are now accurate.
- Because reporting was delayed, this issue also led to problems with selecting a winner for A/B tests sent between 11am and 2pm ET yesterday. All emails sent via A/B tests were delivered. However, for some campaigns selecting the winning variation was delayed up to 5 hours. A few A/B campaigns did not have engagement data a few hours into the test and so a winning variation was chosen randomly to send the rest of the campaign.
- And on the segments side, increased event processing caused intermittent segmentation delays between 6pm and 9pm ET yesterday. This issue was resolved by adding additional processing capacity. But that increased load on our databases, which caused some users to experience app timeouts between 6pm and 7:30pm ET.
None of the event processing delays caused issues with flows, prebuilt segments, or campaign sending.
I’m confident that the updates we’ve made in the last 24 hours will set us up to better support you throughout the rest of the holiday season. Our goal is to provide you with the best possible experience. If you have any questions, please don’t hesitate to get in touch — you can reach me directly at firstname.lastname@example.org.
On behalf of myself and every Klaviyo employee, thank you for being a part of our community.