Overview

The data export feature provides the capability to bulk export your CleverTap event data to your AWS S3 bucket. You can use this for analysis in BI tools or for storage in your data warehouse for analysis in the future.

πŸ“˜

Feature Part of the CleverTap for Enterprises Plan

This capability is a part of the CleverTap for Enterprises plan. Contact your sales account manager to have it activated for you.

Initiate an Export

To initiate an export, perform the following steps:

  1. Navigate to Settings > Partners > Exports.
  2. Click + Create Export.

This feature can be configured to export data for all events or specific events you select.

  1. If required, set up automatic recurring exports ranging from every four hours to every 24 hours.
  2. Click Create export.

Setup Steps

This section covers the setup steps.

Create an AWS S3 Bucket

Depending on your CleverTap account settings, we host your data in Europe (EU), United States (US), Singapore (SG), or India (IN).

Once you create an account in the same region as your CleverTap data hosting region, the next step is to create your S3 bucket in that same region.

πŸ‘

Region Example

If your CleverTap account uses AWS EU (Ireland), then you will have to create an S3 bucket in the AWS EU (Ireland) region.

  1. Log in to your AWS account, then search for S3 in the AWS services box.
  2. Click S3 in the search results.
  1. Click the + Create bucket button.
  1. Enter a bucket name, region, and other settings you may need to configure.

  2. Complete your setup by selecting your logging, versioning, and encryption preferences. For this integration, you do not need to make any modifications to the default settings; however, you should check your internal organization's policies to see if you need to modify any of these settings.

  3. Click Next.

  1. On this page, you can keep the default settings. Again, make sure to check your internal organization's policies to see if you need to modify any of these settings.

  2. Click Next.

  1. Click the Create bucket button.

The bucket you just created should now show up in your S3 console. Note the name of your new bucket since you will need it for the next step below.

Create an API Key for Your S3 Bucket

This step demonstrates how to create an AWS API key that has write access to the bucket we created in the previous step above. CleverTap uses this API key to export data to your S3 bucket.

  1. Click on your account name on the top right of the AWS console.

  2. Click My Security Credentials.

  1. Click Users, then click the Add user button.
  1. Enter a name for the new user and check the Programmatic access checkbox.
  2. Click Next.
  1. Click Add user to group, then click Create group.
  1. Enter a Group name, then click Create policy.
  1. Click on the JSON tab.
  1. Paste the JSON below in the box. Replace "clevertap-example" with the name of the S3 bucket you created in the previous step above.

The permissions defined in this policy will allow CleverTap to get information about your bucket and upload files to it.

πŸ“˜

AWS API Access Policies

IP Whitelisting is not supported with S3 exports. For more information about AWS API access policies, please review this post from the AWS blog.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::clevertap-example"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::clevertap-example/*"
            ]
        }
    ]
}
  1. Click the Review policy button.
  1. Enter a name. Then, click Create policy.
  1. Go back to the Group page. Search for the policy you just created, assign it to the new group by selecting the checkbox next to it.
  2. Click Create group.
  1. Now, add the user to the group. Then, click on the Next Review button.
  1. Click Create user.

On the next page, you will be shown your Access key ID and the Secret access key.

These credentials will allow CleverTap to upload files to your S3 bucket. Note these values since you will use them in the next step below.

Add Your S3 Bucket Details to CleverTap

To add your S3 bucket details to Clevertap, perform the following steps:

  1. Log in to your CleverTap account.
  2. Navigate to Settings > Partners > Exports.
  3. Click the AWS Amazon icon.
  4. Enter your bucket name from the Create an AWS S3 Bucket section above.
  5. Enter your Access key ID and the Secret access key from the Create an API Key for Your S3 Bucket section.

Create a New Data Export

To create a new data export, perform the following steps:

  1. Navigate to Settings > Partners > Exports > Activity Log.
  2. Click + Create Export.

When you create an export, you must define various settings:

Choose Export Partner

  • AWS for exporting data to AWS.

Choose Export Type

  • All user events: This exports data for all events that have been defined which include System and Custom events.
  • Select events: This exports specific events you want to export.
  • All user profiles: This exports all your user profile data.

Choose Export Frequency

  • One time: Single export for the export type selected. You can export data up to the last 45 days.
  • Recurring: Setup a recurring export which exports all the new events/user profiles captured in the last window. You can export as frequently as every 4 hours and up to once every 24 hours.

Choose Export Format

  • JSON
  • XML
  • CSV
  • Parquet

If there is a scheduled recurring profile export, you must stop it to run a one-time profile export.

CleverTap will now process the export. You can refresh the Activity Log page to see the current status of the export.

Once the export is completed, the status for that export will say Done. For recurring exports, the status will be in Pending.

To confirm your event data was successfully exported:

  1. Log in to your AWS account and navigate to the S3 console. You should see some files uploaded to your S3 bucket as shown in the screenshot below.

  2. Copy the request ID from the activity log and search with that in S3. You should see your respective file there.

Export Format

Below is a list of the various export formats.

File Name Format

The example below shows the file name format for data exports:

  • The export request ID is generated when you create a request in the CleverTap dashboard.
  • The timestamp is when the export was run.
  • The event name is an event type that is included in the file.
  • For larger exports, we chunk the data across multiple files. The file index notes what file number in the file series it is. We have limited file sizes to 100 MB chunks to make them more consumable.
<export request id>-<timestamp of the export run>-<event name>-<yyyymmdd>-<file index>-<database-id>.json

File Data Format

Files are split by event names for event exports and each file will have all event data for the given period for the event.

JSON

The first line of the file contains the event name. After the first line, each line in JSON describes the timestamp, object id, and event properties.

{
    "profile": {
        "identity": "dqsndckfk234"
    },
    "ts": 20171109000015,
    "eventProps": {
        "ct_connected_to_wifi": "false",
        "ct_bluetooth_version": "ble",
        "ct_bluetooth_enabled": "false",
        "ct_sdk_version": 30107,
        "ct_latitude": -6.1975594,
        "ct_longitude": 106.52913,
        "ct_os_version": "5.1.1",
        "ct_app_version": "2.30.1",
        "ct_network_carrier": "3",
        "ct_network_type": "4G"
    }
}

CSV

CSV files are comma-delimited and have each event in separate rows.

XML

XML have the timeStamp, eventName, followed by eventProperties.

<Event>
    <ts>20200220130735</ts>
    <eventName>Export Custom Event</eventName>
    <profile>
        <all_identities>[email protected]</all_identities>
        <platform>Web</platform>
        <email>[email protected]</email>
    </profile>
    <deviceInfo>
        <browser>Others</browser>
    </deviceInfo>
    <eventProps>
        <entry>
            <key>CT Source</key>
            <value>API</value>
        </entry>
        <entry>
            <key>Category</key>
            <value>Mens Watch</value>
        </entry>
        <entry>
            <key>Product name</key>
            <value>Casio Chronograph Watch</value>
        </entry>
        <entry>
            <key>Price</key>
            <value>59.99</value>
        </entry>
        <entry>
            <key>Currency</key>
            <value>USD</value>
        </entry>
    </eventProps>
</Event>

Parquet

Parquet have timestamp, eventName, and eventProperties for each event.

πŸ“˜

Parquet File Format

Parquet is an open source file format for Hadoop. Parquet stores nested data structures in a flat columnar format.

Updated 3 months ago


AWS S3


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.