AWS S3
Overview
The data export feature provides the capability to bulk export your CleverTap event data to your AWS S3 bucket. You can use this for analysis in BI tools or storage in your data warehouse for analysis in the future.
Setup
The following are the two major steps involved in enabling this feature for your account:
- Create an AWS S3 Bucket.
- Configure Buckets details on the CleverTap Dashboard using one of the following methods:
Create an AWS S3 Bucket
To create an AWS S3 bucket:
- Log in to your AWS account, then search for S3 in the AWS services box.
- Select S3 from the search results.

Select S3 from AWS Account
On clicking, the Bucket page displays.
- Click Create bucket. On clicking, the Create bucket page displays:

Click Create Bucket
- Enter a bucket name, region, logging, versioning, and encryption preferences.

Configure S3 Bucket Settings and Click Create Bucket
Recommendation for Setup
For this integration, you need not modify the default settings; however, you must check your internal organization's policies to verify if you need to modify any of these settings.
Based on your CleverTap account settings, we host your data in Europe (EU), the United States (US), Singapore (SG), or India (IN). To identify the region of your account and the corresponding region that you must select when configuring the bucket settings, refer to the following table:
CleverTap Dashboard URL | Region | AWS S3 Bucket Region |
---|---|---|
https://eu1.dashboard.clevertap.com/login.html | EU | EU (Ireland) eu-west-1 |
https://in1.dashboard.clevertap.com/login.html | India | Asia Pacific (Mumbai) ap-south-1 |
https://us1.dashboard.clevertap.com/login.html | US | US West (Oregon) us-west-2 |
https://sg1.dashboard.clevertap.com/login.html | Singapore | Asia Pacific (Singapore) ap-southeast-1 |
https://sk1.dashboard.clevertap.com/login.html | South Korea | Asia Pacific (Seoul)\nap-northeast-2 |
- Click Create bucket. On successful bucket creation, the following message displays in the snack bar at the top:

S3 Bucket Created Successfully
The bucket you just created now shows up on your S3 console. We recommend you note down the name of your new bucket, as you will need it for the next step.
Configure S3 Bucket Details on CleverTap Dashboard
Using IAM Policy
The following are the two major steps to perform this task:
i. Copy IAM Policy from the CleverTap dashboard
ii. Add IAM Policy to S3 Bucket from AWS console.
Copy IAM Policy from the CleverTap Dashboard
To copy the IAM policy from the CleverTap dashboard:
- Navigate to Settings > Partners and click View Details against Amazon S3. The Integrate analytics partner - Amazon S3 window displays on the right side of the screen.

Click View Details against Amazon S3 from Partner Page
- Click + Amazon S3 Bucket to create a new bucket.
- Enter the Bucket name.
- Select IAM (Identity and Access Management) Policy option under Configure with section.

Configure S3 Bucket with IAM Policy
- Click the
icon to copy the policy to the clipboard and keep this window open for saving these details later.
Add IAM Policy to S3 Bucket on AWS Console
To add IAM policy to the S3 bucket:
- Select the bucket from the Bucket page of the AWS console.
- Select the Permissions tab.
- Click Edit under the Bucket policy section and enter the policy that you copied in Step 4 of Copy IAM Policy from the CleverTap dashboard:

Edit S3 Bucket Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::062484260092:root"
},
"Action": "s3:PutObject*",
"Resource": "arn:aws:s3:::bucket-name/*"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::062484260092:root"
},
"Action": "s3:PutObject*",
"Resource": "arn:aws:s3:::bucket-name"
}
]
}
IMPORTANT
Ensure that you replace all the occurrences of
bucket-name
in the above JSON payload with your actual bucket name.
- Click Save Changes to save the policy.
- Go back to the same window opened in Step 4 of Copy IAM Policy From CleverTap Dashboard and click Save Credentials.
Amazon S3 bucket is now configured on both the AWS S3 console and CleverTap dashboard.
Using Access Key
The following are the two major steps to perform this task:
i. Create an API Key for Your S3 Bucket.
ii. Add Your S3 Bucket Details to CleverTap.
Create an API Key for Your S3 Bucket
This section demonstrates the creation of an AWS API key with write access to the bucket we created in the above step. CleverTap uses this API key to export data to your S3 bucket.
- Click on your account name on the top right of the AWS console.
- Select Security credentials.

Select Security credentials
- Select Users from the left navigation and click Add user.

Add Users
On clicking, the Add user page displays.
- Enter the User name and select the Programmatic access checkbox.
- Click Next:Permissions.

Select AWS Access Type and Click Next: Permissions
On clicking, Set permissions page displays.
- Click Create group under Add user to group tab.

Add User to Group
On clicking, Create group page displays.

Assign Policy to Group
- Click Create policy. On clicking, Create policy page opens in a new tab.
- Select the JSON tab and then paste the JSON code given below in the box.
- Replace
clevertap-example
in the JSON code with the name of the S3 bucket you created in the above step.
The permissions defined in this policy allow CleverTap to get information about your bucket and upload files to it.
AWS API Access Policies
IP Whitelisting is not supported with S3 exports. For more information about AWS API access policies, refer to this post from the AWS blog.

Bucket Policy in JSON Format
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::clevertap-example"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::clevertap-example/*"
]
}
]
}
- Click Next: Tags and then click Next: Review. On clicking, the Create policy page opens.
- Enter the policy Name and click Create policy.

Enter Policy Name and Click Create policy
On successful policy creation, the following message displays in the snack bar at the top:

Bucket Policy Created Successfully
- Go back to the Create group page (opened in Step 6), search for the policy you just created, and assign it to the new group by selecting the checkbox.
- Click Create group. On clicking, the Add user page displays.

Select Policy and Click Create Group
- Click Next: Tags and then click Next: Review.

Click Next: Tags
- Click Create user.

Click Create user
On successful user creation, the following page displays. You will see your Access key ID and the Secret access key.

Copy or Download User Security Credentials
These credentials allow CleverTap to upload files to your S3 bucket. We recommend you note down these values as you will require them for the next step. Otherwise, you can also click Download .csv to save these details for future use.
Add Your S3 Bucket Details to CleverTap
To add your S3 bucket details to Clevertap, perform the following steps:
- Navigate to Settings > Partners and click View Details against Amazon S3. The Integrate analytics partner - Amazon S3 window displays on the right side of the screen.

Click View Details against Amazon S3 from Partners Page
- Click + Amazon S3 Bucket to create a new bucket. The Integrate Amazon S3 Bucket window opens on the right side of the screen.
- Enter the Bucket nickname and Bucket URL.
- Select the User details option under Configure with section.
- Enter your Access key ID and the Secret access key obtained earlier and click Save Credentials.

Enter User Security Credentials and Click Save Credentials
Create a New Data Export
To create a new data export, perform the following steps:
- Navigate to Settings > Partners > Exports.
- Click + Export and select Amazon S3 from the Partners dropdown.

Create Export
On clicking, the Export to Amazon S3 popup displays on the right side of the screen.

Enter Export Details
-
Configure the following settings:
- Type: Select the events to export from the available options. For more information, refer to Export Details.
- Frequency: Select from one of the following options:
- Once only: A single export for the selected export type. You can export data up to the last 60 days. You create an export for a specific day, date range, previous month, current month, and more.
- Recurring: Set up a recurring export that exports all the new events captured in the last window. You can export data as frequently as every 4 hours and up to once every 24 hours.
-
Click Create export. On clicking, the popup closes, and the following message displays at the top of the Exports page:

Amazon S3 Export Initiated
CleverTap processes the export, and you can now see the newly created export for AWS S3.

New Amazon S3 Export Displays on Exports Page
The status for each export is displayed as Pending as soon as the export is created. The status changes to Running after the processing starts. And, it changes to Done when the export is complete.
Stop Export
You can also stop the export that you have created. To do so, click the
icon for the export request you want to stop, and then click Stop to confirm your action.
- Confirm if your event data was successfully exported. To do so:
a. Log in to your AWS account and navigate to the S3 console.
b. Search your Bucket from the Buckets page and then click the bucket name.
c. Copy the Request ID from the activity log and search with that ID. You should see your respective file there.

New Amazon S3 Export Displays on Amazon S3 Dashboard
Export Details
Export Type
- All user events: This exports data for all events that have been defined, which include System and Custom events.
- Select events: This exports specific events you want to export.
- All user profiles: This exports all your user profile data.
Export Frequency
- One time: Single export for the export type selected. You can export data up to the last 60 days.
- Recurring: Set up a recurring export that exports all the new events/user profiles captured in the last window. You can export as frequently as every 4 hours and up to once every 24 hours.
Export Format
- JSON
- XML
- CSV
- Parquet
For more information, refer to the following section.
Export Format
This section provides information about the file format and the name format of the files exported to the S3 bucket.
File Name Format
- File Name Format for Event Export
The example below shows the file name format for event export:- Export request ID: Indicates the export request ID generated when you create a request in the CleverTap dashboard.
- Timestamp of export run: Indicates when the export was run.
- Event name: Indicates the event type that is included in the file.
- File index: We chunk the data across multiple files for larger exports. We limit file sizes to 100 MB chunks to make them more consumable. The file index indicates the file number in the file series.
- Database ID: Indicates the database ID of the CleverTap from where the file was exported.
- File format: Indicates the format of the file exported to the S3 bucket.
<export request id>-<timestamp of the export run>-<event name>-<yyyymmdd>-<file index>-<database-id>.json
- File Name Format for User Profile Export:
The example below shows the file name format for user profile export:- Account ID: Indicates the integer value for your CleverTap project ID.
- Request ID: Indicates the export request ID generated when you create a request in the CleverTap dashboard.
- Timestamp of export run: Indicates when the export was run.
- Database ID: Indicates the database ID of the CleverTap from where the file was exported.
- File format: Indicates the format of the file exported to the S3 bucket.
<account id>-<request id>-<timestamp of the export run>-<database-id>-<file format>.gz
File Data Format
Files are split by event names for event exports and each file will have all event data for the given period for the event.
JSON
The first line of the file contains the event name. After the first line, each line in JSON describes the timestamp, object id, and event properties.
{
"profile": {
"identity": "dqsndckfk234"
},
"ts": 20171109000015,
"eventProps": {
"ct_connected_to_wifi": "false",
"ct_bluetooth_version": "ble",
"ct_bluetooth_enabled": "false",
"ct_sdk_version": 30107,
"ct_latitude": -6.1975594,
"ct_longitude": 106.52913,
"ct_os_version": "5.1.1",
"ct_app_version": "2.30.1",
"ct_network_carrier": "3",
"ct_network_type": "4G"
}
}
CSV
CSV files are comma-delimited and have each event in separate rows.

Sample CSV File
XML
XML has the timeStamp, eventName, followed by eventProperties.
<Event>
<ts>20200220130735</ts>
<eventName>Export Custom Event</eventName>
<profile>
<all_identities>[email protected]</all_identities>
<platform>Web</platform>
<email>[email protected]</email>
</profile>
<deviceInfo>
<browser>Others</browser>
</deviceInfo>
<eventProps>
<entry>
<key>CT Source</key>
<value>API</value>
</entry>
<entry>
<key>Category</key>
<value>Mens Watch</value>
</entry>
<entry>
<key>Product name</key>
<value>Casio Chronograph Watch</value>
</entry>
<entry>
<key>Price</key>
<value>59.99</value>
</entry>
<entry>
<key>Currency</key>
<value>USD</value>
</entry>
</eventProps>
</Event>
Parquet
Parquet has a timestamp, eventName, and eventProperties for each event.
Parquet File Format
Parquet is an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format.
FAQs
What should I do if I already have an existing IAM policy for the AWS S3 bucket?
Ans. Ensure that you modify the current IAM policy to provide CleverTap with write access to export data to your AWS S3 bucket.

IAM Policy to Provide CleverTap with Write Access
Updated 1 day ago