Google Cloud Platform (GCP)
What's New
CleverTap now supports adding multiple Google Cloud buckets and exporting data to respective buckets on the Google Cloud Platform (GCP).
Overview
The GCP Export feature enables you to bulk export your CleverTap event or profile data to a GCP bucket. You can export your CleverTap data to analyze it with Business Intelligence (BI) tools or store it in your data warehouse for future analysis.
Note
For more information and enable this feature for your account, contact your account manager at CleverTap.
Integrate GCP with CleverTap
The integration involves the following steps:
Create a Service Account for your Project
To create a service account:
- Log in to your GCP account and select the project where you want to export your CleverTap data. Create a new project if you do not have a project.
- From the left panel, select IAM and then navigate to Admin > Service Accounts.
- Create a new Service Account linked to the selected project and enter the Service account name and description, respectively.
This is the account that you will use to authenticate and authorize CleverTap to upload exports to your GCP project.

Create a New Service Account
- Navigate to Permissions > Storage Admin and continue.

Navigate to Storage Admin
- Click + Create Key and select JSON for the Service account.
The JSON key downloads to your machine automatically. Save this key because you will have to upload the key to the CleverTap dashboard later.

Create Key
Create a Bucket
Create a bucket that will hold the export files from CleverTap. To do so:
- From the left panel, navigate to Storage and click Create Bucket.

Navigate to Storage Browser from GCP Dashboard
On clicking, Create a bucket page opens.
- Name your bucket and fill in the other details as per your requirements.

Enter your Bucket Details and Create a Bucket
- Click Create. Following is an image for a successfully created bucket:

Bucket Created Successfully
Configure CleverTap Dashboard
Add the Service account credentials (created in Step 1 and bucket name to CleverTap. To configure the dashboard:
- Navigate to Settings > Partners from the CleverTap dashboard.

Navigating to Partners
- Hover on the Google Cloud icon and click Integrate.

Integrate Google Cloud
The Integrate analytics partner - Google Cloud window opens on the right side of the screen.
- Click + Google Cloud Bucket to add a bucket. You can also add multiple buckets from this popup.
- Enter the following details and click Save Credentials:
Field | Description |
---|---|
Bucket nickname | Enter the nickname for your bucket. |
Service Account Key | Enter the contents of the downloaded Service Account Key JSON file (obtained in Step 1) into the Service Key. |
Bucket name | Add the name of the bucket as given in the project. |

Enter Google Cloud Bucket Details
Error When Saving Credential
- Check if the bucket name is the same as in GCP. Bucket names need to be an exact match.
- Generate the Service Account key again, copy and paste the new key into the Service Account Key field on the Integrate analytics partner - Google Cloud window.

Service Account Key
After successful integration, the Integrated tag displays against Google Cloud on the Partner List page.

Google Cloud Integration Successful
Create a New Data Export
To create a new data export:
- Navigate to Settings > Partners > Exports from the CleverTap dashboard.
- Click + Export and select Google Cloud from the partners dropdown.

Create Export
The Export to Google Cloud window displays on the right side of the screen.

Enter Export Details
-
Configure the following settings:
-
Type: Select the events from the available options to export.
- All events: Export data for all events that have been defined, that is, system and custom events.
- Select events: Exports only specific events.
- All user properties: Exports all your user profile data.
-
Frequency: Select the frequency of export.
- Once only: A single export for the selected export type. You can export data up to the last 60 days. You create an export for a specific day, date range, previous month, current month, and more.
- Recurring: Set up a recurring export to export all the new events or user profiles captured in the last window. You can export as frequently as every 4 hours and up to once every 24 hours.
-
Format: Select from the following export formats: JSON, XML, CSV, and Parquet. For more information, refer to Export Format.
-
Stop Recurring Profile Export to do One Time Export
Before running a one-time profile export, you must stop any scheduled recurring profile export.
- Click Export. On clicking, the popup closes, and the Google export has initiated message displays at the top of the Exports page. CleverTap processes the export, and you can now see the newly-created export for Google Cloud.

Google Export Initiated
To confirm that your event data was successfully exported, select Storage from the left panel of Google Cloud. Navigate to the Bucket details page and search with the requestid
(available under the Exports page of the CleverTap dashboard). All the exported files are displayed.

New Google Export Displays on Google Cloud
Export Format
This section provides information about the file format and the name format of the files being exported to the Google Cloud bucket.
File Name Format
- File Name Format for Event Export
The example below shows the file name format for event export:- Export request ID: Indicates the export request ID generated when you create a request in the CleverTap dashboard.
- Timestamp of the export run: Indicates when the export was run.
- Event name: Indicates the event type that is included in the file.
- File index: We chunk the data across multiple files for larger exports. We limit file sizes to 100 MB chunks to make them more consumable. The file index indicates the file number in the file series.
- Database ID: Indicates the database ID of the CleverTap from where the file was exported.
- File format: Indicates the format of the file exported to the GCP bucket.
<export request id>-<timestamp of the export run>-<event name>-<yyyymmdd>-<file index>-<database-id>.json
- File Name Format for User Profile Export:
The example below shows the file name format for user profile export:- Account ID: Indicates the integer value for your CleverTap project ID.
- Request ID: Indicates the export request ID generated when you create a request in the CleverTap dashboard.
- Timestamp of the export run: Indicates when the export was run.
- Database ID: Indicates the database ID of the CleverTap from where the file was exported.
- File format: Indicates the format of the file exported to the GCP bucket.
<account id>-<request id>-<timestamp of the export run>-<database-id>-<file format>.gz
File Data Format
Export files are split by event names and date range for the specified event.
JSON
The first line of the file contains the event name. After the first line, each line is JSON describing the timestamp, object id, and event properties.
{
"profile": {
"identity": "dqsndckfk234"
},
"ts": 20171109000015,
"eventProps": {
"ct_connected_to_wifi": "false",
"ct_bluetooth_version": "ble",
"ct_bluetooth_enabled": "false",
"ct_sdk_version": 30107,
"ct_latitude": -6.1975594,
"ct_longitude": 106.52913,
"ct_os_version": "5.1.1",
"ct_app_version": "2.30.1",
"ct_network_carrier": "3",
"ct_network_type": "4G"
}
}
CSV
CSV files are comma-delimited and have each event in separate rows.
XML
XML files have the timeStamp, and eventName, followed by eventProperties.
<Event>
<ts>20200220130735</ts>
<eventName>Export Custom Event</eventName>
<profile>
<all_identities>[email protected]</all_identities>
<platform>Web</platform>
<email>[email protected]</email>
</profile>
<deviceInfo>
<browser>Others</browser>
</deviceInfo>
<eventProps>
<entry>
<key>CT Source</key>
<value>API</value>
</entry>
<entry>
<key>Category</key>
<value>Mens Watch</value>
</entry>
<entry>
<key>Product name</key>
<value>Casio Chronograph Watch</value>
</entry>
<entry>
<key>Price</key>
<value>59.99</value>
</entry>
<entry>
<key>Currency</key>
<value>USD</value>
</entry>
</eventProps>
</Event>
Parquet
Parquet has a timestamp, eventName, and eventProperties for each event.
Parquet File Format
Parquet is an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format.
Updated 2 months ago