Overview

The data export feature provides the capability to bulk export your CleverTap event data to your AWS S3 bucket. You can use this for analysis in BI tools or storage in your data warehouse for analysis in the future.

Setup

The following are the two major steps involved in enabling this feature for your account:

1. Create an AWS S3 Bucket.
2. Configure Buckets details on the CleverTap Dashboard using one of the following methods:
   • Using Access Key
   • Using IAM Policy

Create an AWS S3 Bucket

To create an AWS S3 bucket:

1. Log in to your AWS account, then search for S3 in the AWS services box.
2. Select S3 from the search results.

On clicking, the Bucket page displays.

3. Click Create bucket. On clicking, the Create bucket page displays:

4. Enter a bucket name, region, logging, versioning, and encryption preferences.

📘

Recommendation for Setup

For this integration, you need not modify the default settings; however, you must check your internal organization's policies to verify if you need to modify any of these settings.

Based on your CleverTap account settings, we host your data in Europe (EU), the United States (US), Singapore (SG), or India (IN). To identify the region of your account and the corresponding region that you must select when configuring the bucket settings, refer to the following table:

5. Click Create bucket. On successful bucket creation, the following message displays in the snack bar at the top:

The bucket you just created now shows up on your S3 console. We recommend you note down the name of your new bucket, as you will need it for the next step.

Configure S3 Bucket Details on CleverTap Dashboard

Using IAM Policy

The following are the two major steps to perform this task:
i. Copy IAM Policy from the CleverTap dashboard
ii. Add IAM Policy to S3 Bucket from AWS console.

Copy IAM Policy from the CleverTap Dashboard

To copy the IAM policy from the CleverTap dashboard:

1. Navigate to Settings > Partners and click View Details against Amazon S3. The Integrate analytics partner - Amazon S3 window displays on the right side of the screen.

2. Click + Amazon S3 Bucket to create a new bucket.
3. Enter the Bucket name.
4. Select IAM (Identity and Access Management) Policy option under Configure with section.

4. Click the icon to copy the policy to the clipboard and keep this window open for saving these details later.

Add IAM Policy to S3 Bucket on AWS Console

To add IAM policy to the S3 bucket:

1. Select the bucket from the Bucket page of the AWS console.
2. Select the Permissions tab.
3. Click Edit under the Bucket policy section and enter the policy that you copied in Step 4 of Copy IAM Policy from the CleverTap dashboard:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::062484260092:root"
            },
            "Action": "s3:PutObject*",
            "Resource": "arn:aws:s3:::bucket-name/*"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::062484260092:root"
            },
            "Action": "s3:PutObject*",
            "Resource": "arn:aws:s3:::bucket-name"
        }
    ]
}

🚧

IMPORTANT

Ensure that you replace all the occurrences of bucket-name in the above JSON payload with your actual bucket name.

4. Click Save Changes to save the policy.
5. Go back to the same window opened in Step 4 of Copy IAM Policy From CleverTap Dashboard and click Save Credentials.
   Amazon S3 bucket is now configured on both the AWS S3 console and CleverTap dashboard.

Using Access Key

The following are the two major steps to perform this task:
i. Create an API Key for Your S3 Bucket.
ii. Add Your S3 Bucket Details to CleverTap.

Create an API Key for Your S3 Bucket

This section demonstrates the creation of an AWS API key with write access to the bucket we created in the above step. CleverTap uses this API key to export data to your S3 bucket.

1. Click on your account name on the top right of the AWS console.
2. Select Security credentials.

3. Select Users from the left navigation and click Add user.

On clicking, the Add user page displays.

4. Enter the User name and select the Programmatic access checkbox.
5. Click Next:Permissions.

On clicking, Set permissions page displays.

6. Click Create group under Add user to group tab.

On clicking, Create group page displays.

7. Click Create policy. On clicking, Create policy page opens in a new tab.
8. Select the JSON tab and then paste the JSON code given below in the box.
9. Replace clevertap-example in the JSON code with the name of the S3 bucket you created in the above step.
   The permissions defined in this policy allow CleverTap to get information about your bucket and upload files to it.

📘

AWS API Access Policies

IP Whitelisting is not supported with S3 exports. For more information about AWS API access policies, refer to this post from the AWS blog.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::clevertap-example"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::clevertap-example/*"
            ]
        }
    ]
}

10. Click Next: Tags and then click Next: Review. On clicking, the Create policy page opens.
11. Enter the policy Name and click Create policy.

On successful policy creation, the following message displays in the snack bar at the top:

12. Go back to the Create group page (opened in Step 6), search for the policy you just created, and assign it to the new group by selecting the checkbox.
13. Click Create group. On clicking, the Add user page displays.

14. Click Next: Tags and then click Next: Review.

15. Click Create user.

On successful user creation, the following page displays. You will see your Access key ID and the Secret access key.

These credentials allow CleverTap to upload files to your S3 bucket. We recommend you note down these values as you will require them for the next step. Otherwise, you can also click Download .csv to save these details for future use.

Add Your S3 Bucket Details to CleverTap

To add your S3 bucket details to Clevertap, perform the following steps:

1. Navigate to Settings > Partners and click View Details against Amazon S3. The Integrate analytics partner - Amazon S3 window displays on the right side of the screen.

2. Click + Amazon S3 Bucket to create a new bucket. The Integrate Amazon S3 Bucket window opens on the right side of the screen.
3. Enter the Bucket nickname and Bucket URL.
4. Select the User details option under Configure with section.
5. Enter your Access key ID and the Secret access key obtained earlier and click Save Credentials.

Create a New Data Export

To create a new data export, perform the following steps:

1. Navigate to Settings > Partners > Exports.
2. Click + Export and select Amazon S3 from the Partners dropdown.

On clicking, the Export to Amazon S3 popup displays on the right side of the screen.

3. Configure the following settings:

   • Type: Select the events to export from the available options. For more information, refer to Export Details.
   • Frequency: Select from one of the following options:
     • Once only: A single export for the selected export type. You can export data up to the last 60 days. You create an export for a specific day, date range, previous month, current month, and more.
     • Recurring: Set up a recurring export that exports all the new events captured in the last window. You can export data as frequently as every 4 hours and up to once every 24 hours.

4. Click Create export. On clicking, the popup closes, and the following message displays at the top of the Exports page:

CleverTap processes the export, and you can now see the newly created export for AWS S3.

The status for each export is displayed as Pending as soon as the export is created. The status changes to Running after the processing starts. And, it changes to Done when the export is complete.

📘

Stop Export

You can also stop the export that you have created. To do so, click the
icon for the export request you want to stop, and then click Stop to confirm your action.

5. Confirm if your event data was successfully exported. To do so:
   a. Log in to your AWS account and navigate to the S3 console.
   b. Search your Bucket from the Buckets page and then click the bucket name.
   c. Copy the Request ID from the activity log and search with that ID. You should see your respective file there.

Export Details

Export Type

• All user events: This exports data for all events that have been defined, which include System and Custom events.
• Select events: This exports specific events you want to export.
• All user profiles: This exports all your user profile data.

Export Frequency

• One time: Single export for the export type selected. You can export data up to the last 60 days.
• Recurring: Set up a recurring export that exports all the new events/user profiles captured in the last window. You can export as frequently as every 4 hours and up to once every 24 hours.

Export Format

• JSON
• XML
• CSV
• Parquet

For more information, refer to the following section.

Export Format

This section provides information about the file format and the name format of the files exported to the S3 bucket.

File Name Format

• File Name Format for Event Export
  The example below shows the file name format for event export:
  • Export request ID: Indicates the export request ID generated when you create a request in the CleverTap dashboard.
  • Timestamp of export run: Indicates when the export was run.
  • Event name: Indicates the event type that is included in the file.
  • File index: We chunk the data across multiple files for larger exports. We limit file sizes to 100 MB chunks to make them more consumable. The file index indicates the file number in the file series.
  • Database ID: Indicates the database ID of the CleverTap from where the file was exported.
  • File format: Indicates the format of the file exported to the S3 bucket.

<export request id>-<timestamp of the export run>-<event name>-<yyyymmdd>-<file index>-<database-id>.json

• File Name Format for User Profile Export:
  The example below shows the file name format for user profile export:
  • Account ID: Indicates the integer value for your CleverTap project ID.
  • Request ID: Indicates the export request ID generated when you create a request in the CleverTap dashboard.
  • Timestamp of export run: Indicates when the export was run.
  • Database ID: Indicates the database ID of the CleverTap from where the file was exported.
  • File format: Indicates the format of the file exported to the S3 bucket.

<account id>-<request id>-<timestamp of the export run>-<database-id>-<file format>.gz

File Data Format

Files are split by event names for event exports and each file will have all event data for the given period for the event.

JSON

The first line of the file contains the event name. After the first line, each line in JSON describes the timestamp, object id, and event properties.

{
    "profile": {
        "identity": "dqsndckfk234"
    },
    "ts": 20171109000015,
    "eventProps": {
        "ct_connected_to_wifi": "false",
        "ct_bluetooth_version": "ble",
        "ct_bluetooth_enabled": "false",
        "ct_sdk_version": 30107,
        "ct_latitude": -6.1975594,
        "ct_longitude": 106.52913,
        "ct_os_version": "5.1.1",
        "ct_app_version": "2.30.1",
        "ct_network_carrier": "3",
        "ct_network_type": "4G"
    }
}

CSV

CSV files are comma-delimited and have each event in separate rows.

XML

XML has the timeStamp, eventName, followed by eventProperties.

<Event>
    <ts>20200220130735</ts>
    <eventName>Export Custom Event</eventName>
    <profile>
        <all_identities>[email protected]</all_identities>
        <platform>Web</platform>
        <email>[email protected]</email>
    </profile>
    <deviceInfo>
        <browser>Others</browser>
    </deviceInfo>
    <eventProps>
        <entry>
            <key>CT Source</key>
            <value>API</value>
        </entry>
        <entry>
            <key>Category</key>
            <value>Mens Watch</value>
        </entry>
        <entry>
            <key>Product name</key>
            <value>Casio Chronograph Watch</value>
        </entry>
        <entry>
            <key>Price</key>
            <value>59.99</value>
        </entry>
        <entry>
            <key>Currency</key>
            <value>USD</value>
        </entry>
    </eventProps>
</Event>

Parquet

Parquet has a timestamp, eventName, and eventProperties for each event.

📘

Parquet File Format

Parquet is an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format.

FAQs

What should I do if I already have an existing IAM policy for the AWS S3 bucket?

Ans. Ensure that you modify the current IAM policy to provide CleverTap with write access to export data to your AWS S3 bucket.