Microsoft Azure

Understand how to export data from CleverTap dashboard to Microsoft Azure.

πŸ“˜

Public Beta

This feature is released in Public Beta. For more information about this feature, contact your Success Manager or the CleverTap support.

Overview

Microsoft Azure, a Microsoft cloud computing platform, offers access to, management of, and development of applications and services through global data centers. The Microsoft Azure data export feature allows bulk export of your CleverTap event data to Azure Blob Storage. You can use this for analysis in BI tools or storage in your data warehouse for analysis in the future.

Setup

The following are the two major steps involved in enabling this feature for your account:

  1. Create a Microsoft Azure Storage Account.
  2. Create a Blob Service Container.
  3. Generate a SAS Token.
  4. Configure Microsoft Azure Blob Container details on the CleverTap Dashboard.

Create a Microsoft Azure Storage Account

To create a Microsoft Azure Storage Account:

  1. Log in to your Microsoft Azure account and select Storage Accounts from the left menu.
  2. Click + Create to create a new storage account.
Select Storage Account from Left Manu

Select Storage Account from Left Manu

  1. Enter the Storage account name, and do not modify the default settings.
Enter Storage Account Name

Enter Storage Account Name

  1. Click Review and click Create. The following screen displays once the storage account is created:
Storage Account Created Successfully

Storage Account Created Successfully

The storage account you created now appears under the Storage accounts page.

Create a Blob Service Container

To create a Blob Service Container:

  1. Navigate to the Blobs menu under the Blob Service section of your storage account.
  2. Create + Container to create a Blob Service Container within the storage account you created in the previous section.
Create a Blob Storage Container

Create a Blob Storage Container

  1. Enter the Name of your Blob Service Container, and do not modify the other default settings.
  2. Click Create.

Generate a SAS Token

A Shared Access Signature (SAS) Token is a Uniform Resource Identifier (URI) that provides limited access to an Azure Storage container. It is used when you want to authorize access to storage account assets within a defined time frame while keeping your storage account key confidential. For more information about SAS, refer to Delegate access by using a SAS Token.

To generate a SAS Token:

  1. Select the Container and click the icon.
  2. Select Generate SAS from the dropdown list. The Generate SAS window opens on the right side of the screen.
Generate SAS Popup

Generate SAS Popup

  1. Enter the following details:
FieldDescription
Signing methodSelect Account key from the Signing method list. The Signing method provides secure delegated access to resources in your storage account. For more information, refer to Types of shared access signatures.
Signing keySelect any one key from the following options:
  • Key 1
  • Key 2
  • .
    Stored access policySelect the Stored access policy you want to define for your storage. When defining a policy, you need to define the following: start time, expiry time, and permissions for the signature. To export data from CleverTap to Microsoft Azure, you need Read and Write permissions for the container.

    When defined on a container, it grants permissions to the container itself or to the blobs that it contains. For more information, refer to Define a stored access policy.
    PermissionsThis is a mandatory field. Select the permission for the request made with the service SAS. To export data from CleverTap to Microsoft Azure, you need Read and Write permissions for the container. If you have already assigned the required permissions to your stored access policy, the same permissions will be assigned to the request made with the service SAS. For more information, refer to Account SAS Permissions by Operation.
    Start and expiry date/timeIndicates the start and end date/time during which the blob SAS is valid. CleverTap recommends setting the expiration time to the maximum duration available. This is because you export any data to the container if the expiration date has passed. For more information, refer to Configure a SAS Expiration Policy.
    Allowed IP addressCleverTap recommends not adding any IP address range in this field. It indicates that Microsoft will accept export requests only from the IP address or range of IP addresses defined under this field. If you still want to add the IP address(es), you must whitelist the IP addresses listed under CleverTap IP Ranges.
    Allowed protocolsAllowing requests over HTTPS only is recommended. This field indicates the protocols permitted for a request made with the service SAS.
    1. Click Generate SAS token and URL.

    Copy the generated SAS token and URL, as you will not be able to access it later.

    Configure Microsoft Azure Blob Container on CleverTap Dashboard

    To add your Microsoft Azure Blob Container to Clevertap:

    1. Navigate to Settings > Partners and click Integrate against Mircosoft Azure. The Integrate analytics partner - Microsoft Azure window displays on the right side of the screen.
    Integrate analytics partner - Microsoft Azure

    Integrate analytics partner - Microsoft Azure

    1. Click + Microsoft Azure to create a new bucket. The Integrate Microsoft Azure Bucket window opens on the right side of the screen.
    2. Enter the Bucket Nickname.
    3. Select the Azure Blob Storage Endpoint.
      • Default: Enter the Blob Storage Account name.
    Default Blob Storage Account on Microsoft Azure Portal

    Default Blob Storage Account on Microsoft Azure Portal

    • Custom: Add a custom domain mapped to your Azure Blob Storage Endpoint. For more information about defining a custom endpoint, refer to Map a Custom Domain.
    1. Enter the container name that stores data on Azure and the SAS token we obtained in Step 4 of the Generate a SAS Token section.
    Integrate Microsoft Azure Bucket

    Integrate Microsoft Azure Bucket

    Create a New Data Export

    To create a new data export, perform the following steps:

    1. Navigate to Settings > Partners > Exports.
    2. Click + Export and select Microsoft Azure from the Partners dropdown.
    Create Export

    Create Export

    On clicking, the Export to Microsoft Azure popup displays on the right side of the screen.

    Enter Export Details

    Enter Export Details

    1. Configure the following settings:

      • Type: Select the events to export from the available options. For more information, refer to Export Details.
        • Export Priority for Event Identity: Select priority for user identities when exporting data. User Identifiers will be exported based on selected priority. For more information, refer to the Prioritize User Identity for Exports section.
      • Frequency: Select from one of the following options:
        • Once only: A single export for the selected export type. You can export data up to the last 60 days. You create an export for a specific day, date range, previous month, current month, and more.
        • Recurring: Set up a recurring export that exports all the new events captured in the last window. You can export data as frequently as every 4 hours and up to once every 24 hours.
    2. Click Create export. On clicking, the popup closes, and the following message displays at the top of the Exports page:

    Microsoft Azure Export Initiated

    Microsoft Azure Export Initiated

    CleverTap processes the export, and you can now see the newly created export for Microsoft Azure.

    New Microsoft Azure Export Displays on Exports Page

    New Microsoft Azure Export Displays on Exports Page

    The status for each export is displayed as Pending as soon as the export is created. The status changes to Running after the processing starts. And it changes to Done when the export is complete.

    πŸ“˜

    Stop Export

    You can also stop the export that you have created. To do so, click the icon for the export request you want to stop, and then click Stop to confirm your action.

    1. Confirm if your event data was successfully exported. To do so:
      a. Log in to your Microsoft Azure account and select the Storage account under Resources.
      b. Click the Blob service link under the Properties tab.
      c. Select the Container where you exported the data.
      d. (Optional) Click the Switch to Acces Key link if you encounter any authorization error after selecting the Container.
    New Export Displays on Microsoft Azure Dashboard

    New Export Displays on Microsoft Azure Dashboard

    Edit an Export

    You might need to modify an export to meet specific business requirements or while waiting for the next run. This section describes editing a Live Data Streaming and Recurring export in the Running and Pending (awaiting next run) state.

    πŸ“˜

    Points to Remember

    • In case of running exports, the new changes will apply to the next run.
    • You cannot edit a One-time export, regardless its status (running, pending, completed, or stopped).
    • You cannot change the export from User Profile to Event and vice-versa.
    • You cannot modify exports marked as Completed or Stopped.
    • Export changes for Live DataStreaming take 10-15 minutes to take effect.

    To edit an export:

    1. On the CleverTap dashboard, go to Partners > Exports.
    Ellipsis Icon to Edit the Export

    Ellipsis Icon to Edit the Export

    1. Click the Ellipsis icon. The Edit and the Stop buttons appear.
    Click the Edit Export Icon

    Click the Edit Export Icon

    1. Click the Edit button. The Overview section appears.
    Edit the Export

    Edit the Export

    1. Edit the export details and click Update.

    Export Details

    Export Type

    • All user events: This exports data for all events that have been defined, which include System and Custom events.
    • Select events: This exports specific events you want to export.
    • All user profiles: This exports all your user profile data.

    Export Frequency

    • One time: Single export for the export type selected. You can export data up to the last 60 days.
    • Recurring: Set up a recurring export that exports all the new events/user profiles captured in the last window. You can export as frequently as every 4 hours and up to once every 24 hours.

    Export Format

    • JSON
    • XML
    • CSV
    • Parquet

    For more information, refer to the Export Format section.

    Export Format

    This section provides information about the file format and the name format of the files exported to the S3 bucket.

    File Name Format

    • File Name Format for Event Export
      The example below shows the file name format for event export:
      • Export request ID: Indicates the export request ID generated when you create a request in the CleverTap dashboard.
      • Timestamp of export run: Indicates when the export was run.
      • Event name: Indicates the event type that is included in the file.
      • File index: We chunk the data across multiple files for larger exports. We limit file sizes to 100 MB chunks to make them more consumable. The file index indicates the file number in the file series.
      • Database ID: Indicates the database ID of the CleverTap from where the file was exported.
      • File format: Indicates the format of the file exported to the S3 bucket.
    <export request id>-<timestamp of the export run>-<event name>-<yyyymmdd>-<file index>-<database-id>.json
    
    • File Name Format for User Profile Export:
      The example below shows the file name format for user profile export:
      • Account ID: Indicates the integer value for your CleverTap project ID.
      • Request ID: Indicates the export request ID generated when you create a request in the CleverTap dashboard.
      • Timestamp of export run: Indicates when the export was run.
      • Database ID: Indicates the database ID of the CleverTap from where the file was exported.
      • File format: Indicates the format of the file exported to the S3 bucket.
    <account id>-<request id>-<timestamp of the export run>-<database-id>-<file format>.gz
    

    File Data Format

    Files are split by event names for event exports, and each file will have all event data for the given period for the event.

    JSON

    The first line of the file contains the event name. After the first line, each line in JSON describes the timestamp, object id, and event properties.

    {
    	"profile": {
    		"identity": "dqsndckfk234"
    	},
    	"ts": 20171109000015,
    	"eventProps": {
    		"ct_connected_to_wifi": "false",
    		"ct_bluetooth_version": "ble",
    		"ct_bluetooth_enabled": "false",
    		"ct_sdk_version": 30107,
    		"ct_latitude": -6.1975594,
    		"ct_longitude": 106.52913,
    		"ct_os_version": "5.1.1",
    		"ct_app_version": "2.30.1",
    		"ct_network_carrier": "3",
    		"ct_network_type": "4G"
    	}
    }
    

    CSV

    CSV files are comma-delimited and have each event in separate rows.

    2032

    Sample CSV File

    XML

    XML has the timeStamp, eventName, followed by eventProperties.

    <Event>
        <ts>20200220130735</ts>
        <eventName>Export Custom Event</eventName>
        <profile>
            <all_identities>[email protected]</all_identities>
            <platform>Web</platform>
            <email>[email protected]</email>
        </profile>
        <deviceInfo>
            <browser>Others</browser>
        </deviceInfo>
        <eventProps>
            <entry>
                <key>CT Source</key>
                <value>API</value>
            </entry>
            <entry>
                <key>Category</key>
                <value>Mens Watch</value>
            </entry>
            <entry>
                <key>Product name</key>
                <value>Casio Chronograph Watch</value>
            </entry>
            <entry>
                <key>Price</key>
                <value>59.99</value>
            </entry>
            <entry>
                <key>Currency</key>
                <value>USD</value>
            </entry>
        </eventProps>
    </Event>
    

    Parquet

    Parquet has a timestamp, eventName, and eventProperties for each event.

    πŸ“˜

    Parquet File Format

    Parquet is an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format.

    Prioritize User Identity for Exports

    The export file includes an identity column with the user's Identity, Phone Number, or Email values. These values are set based on the identities configured in the CleverTap dashboard under the Settings > User Identity page. This feature lets you prioritize the identifier you want to export in the identity column.

    Let us understand how the prioritization works based on the identities selected in the User Identity page:

    • If you select only Identity, export includes the identity value. The export file's identity column is empty if it is unavailable.
    • If you select multiple identifiers, you must set the priorities on the Export page. For instance, you set Priority 1 to Identity and Priority 2 to Email ID. When exporting data, the export prioritizes the Identity value for the identity column. If it is absent, the Email ID is exported under the identity column of the export file. If both are missing, the column remains empty.

    πŸ“˜

    Key Points to Remember

    • If you change the identity later, the export works according to the set priority. To prioritize the modified identities, edit your export.
    • This feature applies only to the following export types: All events and All user properties.
    • For the old running export, this configuration or prirotization is not applicable. You can add the prioritization by editing the running exports.

    To prioritize user identity for exports:

    1. Go to Partners > Exports.
    2. Click Exports > Microsoft Azure.
    3. In the Configuration section, under Export Priority for Event Identity, click + Priority.
    4. Set up the Priority 1, 2, and 3 for the required identities from the drop-down list.
    5. Click Export.
    Prioritize User Identities for Exports

    Prioritize User Identities for Exports

    FAQ

    Q. How to secure the data exports from CleverTap to Microsoft Azure?

    A. You can whitelist IPs to ensure that CleverTap will only have access to push data to your Microsoft Azure bucket. To get the list of whitelisted IP ranges, refer to the CleverTap IP Ranges documentation.

    Q. Do CleverTap data exports allow special characters?

    A. Yes, CleverTap data exports allow the following special characters:

    • CleverTap's export system supports Unicode (UTF-8) character encoding. It facilitates the accurate representation of text in various languages and scripts. For example, Indian regional languages, Arabic, Korean, Russian, Japanese, Chinese, Spanish, Greek, Indonesian, etc.
    • It replaces the following characters with a hyphen to avoid issues in output file generation:
      • Whitespace
      • Tab
      • Slash
      • null (\0)
    • Control characters are replaced with ?. For more information, refer to Control Character.
    • Supports emoji characters; however, some emojis (UTF-16) may not render properly.

    Q. What customer-related errors can stop exports, and how are customers notified when interruptions occur?

    A. Exports can stop due to customer-related errors. For example, invalid/expired credentials or a missing partner bucket. CleverTap emails customers about the issue to ensure timely resolution.

    Here are the customer-related errors that can stop Microsoft Azure export:

    Error CodeError MessageCause and Resolution
    UnknownAzure SAS token is expired.You must create a new Azure SAS token and set it up in CleverTap. For more information, refer to Generate a SAS Token.
    UnknownAzure SAS token is not yet valid.The Azure SAS token is valid on a future date. Ensure you generate a new Azure SAS token with the correct validity start date and set it up in CleverTap. For more information, refer to Generate a SAS Token.

    Exports are checked every hour for failures. If an error occurs, CleverTap sends three emails to the export creator within a span of three days. Emails 1 and 2 include the error details and a link to fix it. It warns that the export will stop if the problem is not fixed within three days. Email 3 notifies the creator that CleverTap has stopped the export.