Overview

Configuring Databricks with CleverTap enables seamless data import, ensuring synchronization and access to relevant information for analysis, personalized engagement, and data-driven growth.

📘

Private Beta

Databricks is a Private Beta release. Databricks integration supports data import into CleverTap. Export capabilities will be available in a future release. Contact your Customer Success Manager for access.

Quick Start Guide for Existing Users

Prerequisites for Integration

If you are setting up Databricks for the first time, ensure you have the following before proceeding with the CleverTap configuration:

• CleverTap Access to configure Databricks.

• Databricks Workspace Details:

  • Host: Workspace domain (for example, adb1234567890123456.17.azuredatabricks.net). Do not include the protocol (https://).
  • HTTP Path: HTTP path of the target SQL Warehouse or endpoint (for example, /sql/1.0/warehouses/abc123def456).
  • Personal Access Token (PAT): Token authorized to access the SQL Warehouse. Ensure the token has an appropriate lifetime (recommended: 90 days or more for production).
  • Catalog and Schema: Unity Catalog objects where data will be read.
  • Port: HTTPS port used to reach Databricks. Defaults to 443 if not specified. Customers behind corporate proxies may need to consult their network administrator for the correct port.

• Databricks Identity and Permissions

  The identity used to generate the PAT (user or service principal) must have, at minimum:

  • USAGE on the selected catalog and schema.
  • SELECT on tables you plan to import (optionally, grant on future tables if your process creates tables later).

📘

Security

Use a dedicated Databricks identity for CleverTap, grant least privilege, and set an expiry/rotation policy for the PAT.

📘

IP Whitelisting
To ensure seamless communication between CleverTap and your Databricks workspace, whitelist the required CleverTap IP ranges in your network and Databricks workspace settings. Refer to CleverTap IP Ranges for the complete list.

Set Up Databricks for Integration

You can set up Databricks using one of the following ways:

• For new users who need to provision Databricks resources from scratch: Create New Databricks Setup
• For users who already have configured resources in Databricks: Use existing Databricks Credentials

Create New Databricks Setup

If you do not already have a Catalog, SQL Warehouse, User/Service Principal, and Schema configured in Databricks, you must create them before proceeding. These components are required to ensure CleverTap can securely access, store, and process your data.

The following table explains each Databricks component and its role in the integration:

To create each resource, perform the following steps:

1. Create a Catalog
2. Create a SQL Warehouse
3. Create Personal Access Token
4. Create a Schema
5. Configure Permissions

Create Catalog

To get your Databricks data in CleverTap, you first need to create a Unity Catalog.

📘

Prerequisite

Creating a catalog requires metastore admin privileges or the CREATE CATALOG permission. Contact your Databricks workspace administrator if you do not have this permission.

Follow these steps:

1. Open a SQL editor in your Databricks workspace with Unity Catalog enabled.
2. Run the following SQL command to create a catalog:
3. SQL

   -- Create a new catalog for CleverTap data storage
   CREATE CATALOG IF NOT EXISTS clevertap_catalog;
   
   -- Verify catalog creation
   SHOW CATALOGS;

Expected Output

+--------------------+
| catalog_name |
+--------------------+
| clevertap_catalog |
+--------------------+

Create SQL Warehouse

CleverTap executes queries through a Databricks SQL Warehouse. Create a dedicated warehouse for CleverTap to avoid contention with other workloads.

To create a SQL Warehouse via the Databricks UI, perform the following steps:

1. In Databricks, navigate to SQL in the left sidebar.
2. Click SQL Warehouses and then click Create.
3. Configure the warehouse with the following recommended settings:
   1. Name: Enter a descriptive name (for example, clevertap_warehouse).
   2. Size: Select Small (sufficient for most CleverTap import workloads).
   3. Auto Stop: Set to 10 minutes to control costs when idle.
   4. Auto Resume: Enable to automatically start the warehouse when a query arrives.
4. Click Create.
5. After the warehouse is created, navigate to the Connection Details tab.
6. Copy the Host and HTTP Path values. You will need these when configuring CleverTap.

📘

Note

Enable Auto Stop to control costs. Databricks charges on a per-second billing model, so setting Auto Stop to 10 minutes ensures the warehouse does not run when idle. Ensure the PAT identity has USE permission on this warehouse.

Create Personal Access Token

Create or identify the Databricks identity you will use with CleverTap, then generate a Personal Access Token (PAT). To generate a PAT, perform the following steps:

1. In Databricks, click your username in the top-right corner of the workspace.
2. Select Settings from the dropdown menu.
3. Under User, select Developer.
4. In the Access Tokens section, click Manage.
5. Click Generate New Token.
6. Enter an optional Comment (for example, CleverTap Integration) to identify the token.
7. Set the Lifetime in days. For production integrations, a minimum of 90 days is recommended. Leave blank for no expiration.
8. Click Generate.
9. Copy the token value immediately and store it securely. The token is displayed only once. You will paste it in the Personal Access Token (PAT) field in CleverTap.

🚧

Token Expiration

PATs expire based on the lifetime you set during creation. When a token expires, the CleverTap connection will fail, and imports will stop. Monitor token expiration dates and regenerate tokens before they expire. Consider setting a calendar reminder.

Create Schema

After creating the catalog, you must create a schema to organize CleverTap-related objects. To do so, perform the following steps:

1. Ensure you are using the correct catalog context:

   SQL

   -- Switch to the CleverTap catalog
   USE CATALOG clevertap_catalog;

2. Execute the following SQL command to create a schema:

   SQL

   -- Create a new schema for CleverTap data
   CREATE SCHEMA IF NOT EXISTS clevertap_schema;
   
   -- Verify schema creation
   SHOW SCHEMAS;

Expected Output

+--------------------+
| schema_name |
+--------------------+
| clevertap_schema |
Databricks | CleverTap Documentation
+--------------------+

Configure Permissions

To ensure the PAT identity (user or service principal) can access the resources created above, grant the required permissions. This step is essential for CleverTap to read data from your Databricks workspace.

Run the following SQL commands, replacing clevertap_user with your actual user or service principal name:

-- Grant catalog-level access
GRANT USAGE ON CATALOG clevertap_catalog TO 'clevertap_user';

-- Grant schema-level access
GRANT USAGE ON SCHEMA clevertap_catalog.clevertap_schema TO 'clevertap_user';

-- Grant read access on all existing tables in the schema
GRANT SELECT ON SCHEMA clevertap_catalog.clevertap_schema TO 'clevertap_user';

📘

Granular Table Permissions

If you prefer to grant access to specific tables rather than the entire schema, use the following command instead: GRANT SELECT ON TABLE clevertap_catalog.clevertap_schema.table_name TO clevertap_user Repeat for each table that CleverTap needs to access.

If the PAT identity also needs access to the SQL Warehouse, ensure the warehouse permissions are configured:

-- Grant SQL Warehouse access (run from warehouse admin context)
GRANT CAN USE ON SQL WAREHOUSE clevertap_warehouse TO 'clevertap_user';

You can verify the grants by running:

SHOW GRANTS ON CATALOG clevertap_catalog;
SHOW GRANTS ON SCHEMA clevertap_catalog.clevertap_schema;

Use Existing Databricks Credentials

If you already have Databricks set up, perform the following steps to find each detail on the Databricks workspace.

• Obtain Host and HTTP Path from SQL Warehouse
• Create or Retrieve a PAT
• Find Existing Catalog
• Find Existing Schema
• Verify Existing Permissions

Obtain Host and HTTP Path from SQL Warehouse

To configure the integration, you need your Databricks Host and the HTTP Path of your SQL Warehouse. Both values are available from the same location. Follow these steps:

1. In Databricks, click SQL in the left sidebar.
2. Select SQL Warehouses from the left panel to view available warehouses.
3. Click on the target warehouse to open its details.
4. Navigate to the Connection Details tab.
5. Copy the Host value (for example, adb-1234567890123456.17.azuredatabricks.net).
6. Copy the HTTP Path value (for example, /sql/1.0/warehouses/abc123def456).

Create or Retrieve a PAT

To authenticate CleverTap with Databricks, generate a Personal Access Token:

1. In Databricks, click your username in the top-right corner.
2. Select Settings > Developer > Access Tokens.
3. Click Generate New Token, enter a comment and lifetime, then click Generate.
4. Copy the token value immediately and store it securely. The token is displayed only once.

📘

Token Rotation

If your organization requires periodic credential rotation, generate a new PAT before the current one expires. Update the CleverTap connection with the new token to avoid import interruptions. Refer to Databricks documentation on Personal Access Tokens for detailed guidance.

Find Existing Catalog

If you need to check for existing catalogs in your workspace:

1. In Databricks, open Data Explorer from the left sidebar.
2. Review the list under Catalogs to identify the catalog to use with CleverTap.
3. Alternatively, run the following SQL command:

SHOW CATALOGS;

Find Existing Schema

To find existing schemas in a specific catalog, perform the following steps:

1. In Data Explorer, select the desired catalog.
2. Select Schemas to view all available schemas within that catalog.
3. Alternatively, run the following SQL command, replacing <catalog_name> with your catalog:

SHOW SCHEMAS IN CATALOG <catalog_name>;

Verify Existing Permissions

Ensure the user or service principal associated with your PAT has the required permissions:

-- Check grants on your catalog
SHOW GRANTS ON CATALOG <catalog_name>;
-- Check grants on your schema
SHOW GRANTS ON SCHEMA <catalog_name>.<schema_name>;

If the required permissions (USAGE and SELECT) are not present, refer to the Configure Permissions section above to grant them.

Set Up CleverTap Dashboard for Integration

You have already prepared your Databricks environment and gathered the required values (Host, HTTP Path, Personal Access Token, Catalog, and optionally Schema and Port). In this section, you will add a Databricks connection in CleverTap, enter these parameters, validate the connection, and proceed to create an import.

Connection Details for Databricks

To connect Databricks with CleverTap, go to Settings > Partners > Databricks and select Add Database. To create or retrieve details from your Databricks workspace, refer to Create a new Catalog, SQL Warehouse, User/Principal, and Schema or Use existing Databricks credentials and configure the following:

Test and Save the Connection

After entering all connection details, complete the setup:

1. Click Test Connection to verify that the workspace, HTTP Path, token, and privileges are correct.
   1. A successful test confirms that CleverTap can reach your Databricks workspace and access the specified catalog and schema.
   2. A failed test indicates an issue with one or more parameters. Refer to the Troubleshooting section below for common errors.
2. Click Save to store the connection details.
3. After saving, navigate to the Import Connections dashboard and click Create Import to set up your first data import.

Troubleshooting

If the Test Connection fails, use the following table to identify and resolve common issues:

FAQs

This section answers common questions related to managing BigQuery connections and imports within CleverTap.

How can I delete a connection that has running imports?

Go to Import Connections, select the connection, click Delete, and review the running imports. If there are any imports in ready, scheduled, paused, or running state, it stops all these active imports associated with the connection. Once done, click Delete and confirm.

How can I filter import connections?

Use the filters on Import Connections to refine the list:

• Connected On: Filter by creation date.
• Connected By: Filter by the user who created the connection.

How can I whitelist IPs for CleverTap integration?

To ensure seamless communication between CleverTap and your systems, whitelist the required IP ranges. To access the list of IPs to whitelist for import integrations, refer to CleverTap IP Ranges.

What happens when my PAT expires?

When the Personal Access Token expires, CleverTap will no longer be able to authenticate with Databricks and all active imports will fail. To resolve this, generate a new PAT in Databricks, then update the CleverTap connection by editing it and replacing the old token with the new one.

Can I use the same Databricks connection for multiple imports?

Yes. A single Databricks connection in CleverTap can be used to create multiple import configurations, each reading from different tables within the same catalog and schema