Solution Recipe 24: AWS S3-Triggered Ingestion via Klaviyo’s SFTP
Solution Recipes are tutorials to achieve specific objectives in Klaviyo. They can also help you master Klaviyo, learn new third-party technologies, and come up with creative ideas. They are written mainly for developers and technically-advanced users.
In this Solution Recipe, we will outline how to connect AWS S3 to Klaviyo’s SFTP to trigger profile ingestion when a new file is added to an S3 bucket. While this recipe covers AWS S3, you can apply this solution to connect any code hosting platform to Klaviyo’s SFTP.
Increasingly, customers need a reliable solution that allows them to effortlessly import a large amount of data. A lot of Klaviyo customers rely on SFTP ingestion to quickly and accurately ingest data. By leveraging AWS, specifically S3 and Lambda, users can trigger ingestion when a new file is added to an S3 bucket, creating a more automatic ingestion schedule that scales for large data sets.
It is time consuming to update and import a large number of profiles (i.e., 500K+). We will outline an AWS-based solution that utilizes an S3 bucket and Klaviyo’s SFTP import tool to reduce the time required to make updates and automate key parts of the process.
When using an S3-triggered SFTP ingestion, you can streamline and automate the process of importing data, which ultimately improves the overall experience of accurately and effectively maintaining customer data and leveraging Klaviyo’s powerful marketing tools. Some use cases that may require this type of solution include a bulk daily sync to update profile properties with rewards data or updating performance results from a third-party integration.
Other key profile attributes you may want to update in bulk and leverage for greater personalization in Klaviyo include:
- Favorite brands
- Properties from off-line sources
In this solution recipe, we’ll walk through:
- Setting up your SFTP credentials
- Configuring your AWS account with the required IAM, S3 and Lambda settings
- Deploying code to programmatically ingest a CSV file into Klaviyo via our SFTP server.
The goal is to streamline and accelerate the process of ingesting profile data into Klaviyo using AWS services and Klaviyo’s SFTP import tool.
- GitHub Link Repo
- Klaviyo SFTP Import tool
- AWS Lambda with S3
- Python Libraries: pandas, pysftp, boto3
We will provide in-depth, step-by-step instructions throughout this Solution Recipe. The more broad overview of steps are:
- Create an SSH key pair and prepare for SFTP ingestion.
- Set up an AWS account with access to IAM, S3, and Lambda
a. Create an IAM execution role for Lambda
b. Create and record security credentials
c. Create an S3 bucket
d. Configure a Lambda function with the IAM execution role and set the S3 bucket as the trigger
e. Configure AWS environment variables.
- Deploy the Lambda function and monitor progress to ensure profiles are updated and displayed in Klaviyo’s UI as anticipated.
Instructions for SFTP configuration
Step 1: Create SSH key pair and prepare for SFTP ingestion
SFTP is only available to Klaviyo users with an Owner, Admin, or Manager role. To start, you’ll need to generate a public/private key pair on your local machine using ssh-keygen or a tool of your choice. Refer to this documentation for the supported SSH key formats.
Once you have generated your keys:
- In Klaviyo, click your account name in the lower left corner and select “Integrations“
- On the Integrations page, click “Manage Sources” in the upper right, then select “Import via SFTP“
- Click “Add a new SSH Key“
- Paste your public key into the “SSH Key” box
- Click “Add key”
Record the following details so you can add them into your AWS Environment Variables in the next step:
- Server: sftp.klaviyo.com
- Username: Your_SFTP_Username (abc123_def456)
- SSH Private Key: the private key associated with the public key generated in previous steps
You can read up on our SFTP tool by visiting our developer portal.
Instructions for AWS implementation
Step 2A. Create an IAM Execution role for Lambda
- Create an IAM role with AWS service as the trusted entity and Lambda as the use-case.
- The Lambda will require the following policy names:
Step 2B. Set up your access key ID and secret access key
You will need to set up your access key ID and secret access key in order to give access to the files to be uploaded from the local machine.
- Navigate to “Users” in the IAM console
- Choose your IAM username
- Open the “Security credentials” tab, and then choose “Create access key”
- To see the new access key, choose “Show”
- To download the key pair, choose “Download .csv file“. Store the file in a secure location. You will add these values into your AWS Environment Variables.
Note: You can retrieve the secret access key only when you create the key pair. Like a password, you can’t retrieve it later. If you lose it, you must create a new key pair.
Step 2C. Create S3 bucket
Navigate here to create the S3 bucket.
Step 2D. Configure a Lambda function
The Lambda is the component of this set-up used to start up the SFTP connection and ingest the CSV file.
- Create a new Lambda function with the execution role configured in step 2A.
- Update the Trigger settings with the S3 bucket created in step 2C.
- Add corresponding files into Lambda from GitHub.
We’ll do a code deep dive in step 3.
Step 2E. Configure AWS environment variables.
- Navigate to the “Configuration” tab and add the following resources into your Environment Variables with their corresponding values.
Step 3: Deploy your code
Let’s review the code deployed in your AWS instance.
- Programmatically add CSV file to S3:
- Download S3 file:
- Prepare CSV file for ingestion:
- Establish SFTP server connection:
- Put it all together in a lambda_handler function to ingest S3 file via SFTP:
Using Klaviyo’s SFTP tool makes data ingestion faster and more efficient. When coupled with the power of AWS’s S3 and Lambda services, you can boost its automation and scalability. With this configuration, your team will be able to manage data and execute ingestion with speed and accuracy, reducing the time and burden of updating profiles manually. Moreover, you can significantly improve data accuracy and mitigate the risk of errors during the ingestion process, ensuring the reliability and integrity of the data you’re using.
Overall, this solution optimizes the efficiency and effectiveness of leveraging relevant data in Klaviyo while streamlining operations and enhancing overall performance.