Overview

Configuring Databricks with CleverTap enables seamless data import, ensuring synchronization and access to relevant information for analysis, personalized engagement, and data-driven growth.

CleverTap supports the following authentication methods for connecting to Databricks:

• Personal Access Token (PAT): Authenticates using a long-lived token generated for a Databricks user or service principal.
• OAuth (Machine-to-Machine): Authenticates as a Databricks service principal using a Client ID and Client Secret. CleverTap automatically exchanges these credentials for short-lived OAuth access tokens. CleverTap recommends using OAuth for production integrations.

Quick Start Guide for Existing Users

Prerequisites for Integration

If you are setting up Databricks for the first time, ensure you have the following before proceeding with the CleverTap configuration:

• CleverTap Access to configure Databricks.

• Databricks Workspace Details:

  • Host: Workspace domain (for example, adb1234567890123456.17.azuredatabricks.net). Do not include the protocol (https://).
  • HTTP Path: HTTP path of the target SQL Warehouse or endpoint (for example, /sql/1.0/warehouses/abc123def456).
  • Authentication Credentials: Depending on the authentication methods you select:
    • Personal Access Token (PAT): Token authorized to access the SQL Warehouse. Ensure the token has an appropriate lifetime (recommended: 365 days or more for production).
    • OAuth Client ID and Client Secret: Credentials generated for a Databricks service principal authorized to access the SQL Warehouse. Verify if the OAuth secret has an appropriate lifetime (recommended: 365 days or more for production; maximum: 730 days).

• Catalog and Schema: Unity Catalog objects where data will be read.

  • Port: HTTPS port used to reach Databricks. Defaults to 443 if not specified. Customers behind corporate proxies may need to consult their network administrator for the correct port.

• Databricks Identity and Permissions

  The identity used to generate the PAT (user or service principal) must have, at a minimum:

  • USE CATALOG on the selected catalog and USE SCHEMA on the selected schema.
  • SELECT on tables you plan to import (optionally, grant on future tables if your process creates tables later).
  • CAN USE on the SQL Warehouse.

Set Up Databricks for Integration

You can set up Databricks using one of the following ways:

• For new users who need to provision Databricks resources from scratch: Create New Databricks Setup
• For users who already have configured resources in Databricks: Use existing Databricks Credentials

Create New Databricks Setup

If you do not already have a Catalog, SQL Warehouse, Personal Access Token, and Schema configured in Databricks, you must create them before proceeding. These components are required to ensure CleverTap can securely access, store, and process your data.

The following table explains each Databricks component and its role in the integration:

To create each resource, perform the following steps:

1. Create Catalog

2. Create SQL Warehouse

3. Create Personal Access Token OR

   Create Service Principal and OAuth Secret

4. Create a Schema

5. Configure Permissions

Create Catalog

To get your Databricks data in CleverTap, you first need to create a Unity Catalog.

To create a Unity catalog, perform the following steps:

1. Open a SQL editor in your Databricks workspace with Unity Catalog enabled.
2. Run the following SQL command to create a catalog:
3. SQL

   -- Create a new catalog for CleverTap data storage
   CREATE CATALOG IF NOT EXISTS clevertap_catalog;
   
   -- Verify catalog creation
   SHOW CATALOGS;

Expected Output

+--------------------+
| catalog_name |
+--------------------+
| clevertap_catalog |
+--------------------+

Create SQL Warehouse

CleverTap executes queries through a Databricks SQL Warehouse. Create a dedicated warehouse for CleverTap to avoid contention with other workloads.

To create a SQL Warehouse via the Databricks UI, perform the following steps:

1. In Databricks, navigate to SQL in the left sidebar.
2. Click SQL Warehouses and then click Create.
3. Configure the warehouse with the following recommended settings:
   1. Name: Enter a descriptive name (for example, clevertap_warehouse).
   2. Size: Select Small (sufficient for most CleverTap import workloads).
   3. Auto Stop: Set to 10 minutes to control costs when idle.
   4. Auto Resume: Enable to automatically start the warehouse when a query arrives.
4. Click Create.
5. After the warehouse is created, navigate to the Connection Details tab.
6. Copy the Host and HTTP Path values. You will need these when configuring CleverTap.

Create Personal Access Token

Create or identify the Databricks identity you will use with CleverTap, then generate a Personal Access Token (PAT). To generate a PAT, perform the following steps:

1. In Databricks, click your username in the top-right corner of the workspace.

2. Select Settings from the dropdown menu.

3. Under User, select Developer.

4. In the Access Tokens section, click Manage.

5. Click Generate New Token.

6. Enter an optional Comment (for example, CleverTap Integration) to identify the token.

7. Set the Lifetime in days. For production integrations, a minimum of 365 days is recommended. The maximum lifetime for a PAT is 730 days.

8. Select scope: sql

   This is the only scope required. CleverTap connects via JDBC (Java Database Connectivity) to your SQL Warehouse and uses standard SQL commands (SHOW CATALOGS, SHOW SCHEMAS, SHOW TABLES, DESCRIBE, SELECT) for all operations, all of which are covered by the sql scope.

9. Ensure the user or service principal the PAT belongs to has the following grants:

   • Unity Catalog grants

     • USE CATALOG on the catalog
     • USE SCHEMA on the schema
     • SELECT on each table or view, CleverTap needs to import

   • Compute access

     CAN USE on the SQL Warehouse, CleverTap will connect to

   📘

   Note

   CleverTap recommends creating a dedicated service principal for this integration and limiting SELECT grants to only the specific tables or views you want CleverTap to import.

10. Click Generate.

11. Copy the token value immediately and store it securely. The token is displayed only once. You will paste it in the Personal Access Token (PAT) field in CleverTap.

Create Service Principal and OAuth Secret

With OAuth (Machine-to-Machine), CleverTap authenticates a Databricks service principal using a Client ID and Client Secret, and automatically exchanges these credentials for short-lived OAuth access tokens. To set this up, perform the following steps:

1. Create Service Principal

   1. In Databricks, click the username in the top-right corner of the workspace and select Settings from the dropdown menu.
   2. Click Identity and access, and click Manage next to Service principals.
   3. Click Add service principal, and then click Add new.
   4. Enter a descriptive name (for example, clevertap-integration) and click Add.

2. Generate OAuth Secret

   1. From the Service principals page, click the service principal you created.
   2. Select the Secrets tab.
   3. Under OAuth secrets, click Generate secret.
   4. Set the secret lifetime in days. For production integrations, we recommend a minimum of 365 days. The maximum lifetime for an OAuth secret is 730 days.

      🚧

      OAuth Secret Expiration

      OAuth Secret expires based on the lifetime you set during creation. When it expires, the CleverTap connection fails, and imports stop. Monitor secret expiration dates and generate a new secret before the current one expires. Consider setting a calendar reminder.

   5. Click Generate.
   6. Copy the displayed Client ID and Secret immediately and store them securely. The secret is displayed only once. Paste these values into the Client ID and Client Secret fields in CleverTap.

      📘

      Client ID

      The Client ID is the same as the Application ID (a UUID) of the Service Principal. You can view it at any time on the Service Principal page. The secret cannot be viewed again after creation; if it is lost, generate a new secret. Each service principal can have up to five OAuth secrets.

3. Grant Required Permissions to Service Principal

   Verify if the Service Principal has the following grants:

   • Unity Catalog grants

     • USE CATALOG on the catalog
     • USE SCHEMA on the schema
     • SELECT on each table or view that CleverTap needs to import

   • Compute access

     • CAN USE on the SQL Warehouse CleverTap will connect to
     • Refer to Configure Permissions below for the SQL commands. When granting privileges to a service principal, reference it by its Application ID.

     📘

     Microsoft Entra ID Service Principal (Azure)

     On Azure Databricks, if your service principal is Microsoft Entra ID managed, you must still generate the OAuth secret in Databricks as described above. A client secret created in Microsoft Entra ID is not the same secret and cannot be used in CleverTap.

Create Schema

After creating the catalog, you must create a schema to organize CleverTap-related objects. To do so, perform the following steps:

1. Ensure you are using the correct catalog context:

   SQL

   -- Switch to the CleverTap catalog
   USE CATALOG clevertap_catalog;

2. Execute the following SQL command to create a schema:

   SQL

   -- Create a new schema for CleverTap data
   CREATE SCHEMA IF NOT EXISTS clevertap_schema;
   
   -- Verify schema creation
   SHOW SCHEMAS;

Expected Output

+--------------------+
| schema_name |
+--------------------+
| clevertap_schema |
+--------------------+

Configure Permissions

To ensure the PAT identity (user or service principal) can access the resources created above, grant the required permissions. This step is essential for CleverTap to read data from your Databricks workspace.

Run the following SQL commands, replacing clevertap_user with your actual user or service principal name:

-- Grant catalog-level access
GRANT USAGE ON CATALOG clevertap_catalog TO 'clevertap_user';

-- Grant schema-level access
GRANT USAGE ON SCHEMA clevertap_catalog.clevertap_schema TO 'clevertap_user';

-- Grant read access on all existing tables in the schema
GRANT SELECT ON SCHEMA clevertap_catalog.clevertap_schema TO 'clevertap_user';

If the identity also needs access to the SQL Warehouse, grant CAN USE from the warehouse permissions settings:

1. In Databricks, navigate to SQL Warehouses and click the target warehouse.
2. Click Permissions.
3. Add the user or service principal and select Can use.

You can verify the grants by running:

SHOW GRANTS ON CATALOG clevertap_catalog;
SHOW GRANTS ON SCHEMA clevertap_catalog.clevertap_schema;

Use Existing Databricks Credentials

If you already have Databricks set up, follow these steps to find each detail in the Databricks workspace.

• Obtain Host and HTTP Path from SQL Warehouse
• Create or Retrieve a PAT
• Create or Retrieve OAuth Credentials
• Find Existing Catalog
• Find Existing Schema
• Verify Existing Permissions

Obtain Host and HTTP Path from SQL Warehouse

To configure the integration, you need your Databricks Host and the HTTP Path of your SQL Warehouse. Both values are available from the same location. Follow these steps:

1. In Databricks, click SQL in the left sidebar.
2. Select SQL Warehouses from the left panel to view available warehouses.
3. Click on the target warehouse to open its details.
4. Navigate to the Connection Details tab.
5. Copy the Host value (for example, adb1234567890123456.17.azuredatabricks.net).
6. Copy the HTTP Path value (for example, /sql/1.0/warehouses/abc123def456).

Create or Retrieve a PAT

To authenticate CleverTap with Databricks, generate a Personal Access Token:

1. In Databricks, click your username in the top-right corner.
2. Select Settings > Developer > Access Tokens.
3. Click Generate New Token, enter a comment and lifetime, then click Generate.
4. Copy the token value immediately and store it securely. The token is displayed only once.

Create or Retrieve OAuth Credentials

This section is required only if you use OAuth authentication. To retrieve the Client ID and generate a Client Secret for an existing service principal, perform the following steps:

1. In Databricks, click the username in the top-right corner, then select Settings > Identity and access.
2. Next to Service principals, click Manage and select the target service principal. If you do not have one, refer to Create a Service Principal and OAuth Secret.
3. Copy the service principal's Application ID. This is the Client ID.
4. Select the Secrets tab and click Generate secret if you do not have an active secret. Existing secret values cannot be retrieved after creation.
5. Copy the secret value immediately and store it securely. The secret is displayed only once.

Find Existing Catalog

If you need to check for existing catalogs in your workspace:

1. In Databricks, open Data Explorer from the left sidebar.
2. Review the list under Catalogs to identify the catalog to use with CleverTap.
3. Alternatively, run the following SQL command:

SHOW CATALOGS;

Find Existing Schema

To find existing schemas in a specific catalog, perform the following steps:

1. In Data Explorer, select the desired catalog.
2. Select Schemas to view all available schemas within that catalog.
3. Alternatively, run the following SQL command, replacing <catalog_name> with your catalog:

SHOW SCHEMAS IN CATALOG <catalog_name>;

Verify Existing Permissions

Ensure the user or service principal associated with your PAT or OAuth credentials has the required permissions:

-- Check grants on your catalog
SHOW GRANTS ON CATALOG <catalog_name>;
-- Check grants on your schema
SHOW GRANTS ON SCHEMA <catalog_name>.<schema_name>;

If the required permissions (USAGE and SELECT) are not present, refer to the Configure Permissions section above to grant them.

Set Up CleverTap Dashboard for Integration

You have already prepared your Databricks environment and gathered the required values (Host, HTTP Path, Personal Access Token, Catalog, and optionally Schema and Port). In this section, you will add a Databricks connection in CleverTap, enter these parameters, validate the connection, and proceed to create an import.

Connection Details for Databricks

To connect Databricks with CleverTap, go to Settings > Partners > Databricks and select Add Database. To create or retrieve details from your Databricks workspace, refer to Create a new Catalog, SQL Warehouse, Personal Access Token, and Schema or Use existing Databricks credentials and configure the following:

Test and Save Connection

After entering all connection details, complete the setup:

1. Click Test Connection to verify that the workspace, HTTP Path, credentials, and privileges are correct.
   1. A successful test confirms that CleverTap can reach your Databricks workspace and access the specified catalog and schema.
   2. A failed test indicates an issue with one or more parameters. Refer to the Troubleshooting section below for common errors.
2. Click Save to store the connection details.
3. After saving, go to the Import Connections dashboard and click Create Import to set up your first data import.

Troubleshooting

If the Test Connection fails, use the following table to identify and resolve common issues:

FAQs

This section answers common questions about managing Databricks connections and imports in CleverTap.

Should I use PAT or OAuth authentication?

For production integrations, CleverTap recommends using OAuth. CleverTap exchanges the Client ID and Client Secret of the Service Principal for short-lived access tokens, and the connection is tied to a dedicated service principal rather than an individual user account. PAT remains fully supported and is a simpler option for trials and quick evaluations.

How can I delete a connection that has running imports?

Go to Import Connections, select the connection, click Delete, and review the running imports. If there are any imports in ready, scheduled, paused, or running state, it stops all these active imports associated with the connection. Once done, click Delete and confirm.

How can I filter import connections?

Use the filters on Import Connections to refine the list:

• Connected On: Filter by creation date.
• Connected By: Filter by the user who created the connection.

How can I whitelist IPs for CleverTap integration?

To ensure seamless communication between CleverTap and your systems, whitelist the required IP ranges. To access the list of IPs to whitelist for import integrations, refer to CleverTap IP Ranges.

What happens when my PAT expires?

When the Personal Access Token expires, CleverTap will no longer be able to authenticate with Databricks, and all active imports will fail. To resolve this, generate a new PAT in Databricks, then update the CleverTap connection by editing it and replacing the old token with the new one.

Can I use the same Databricks connection for multiple imports?

Yes. A single Databricks connection in CleverTap can be used to create multiple import configurations, each reading from different tables within the same catalog and schema