Overview

This document answers common questions about importing user data into CleverTap through SFTP or data warehouses, covering sequencing, data preparation, and best practices to ensure data integrity during historical imports.

What are the different ways to import historical data into CleverTap?

You can upload historical data to CleverTap using any of the following methods, depending on your data volume, frequency, and technical setup:

• SFTP Import: Best for recurring bulk uploads of both user profiles and events. Supports CSV format with manifest file mapping. For more information, refer to SFTP Import.
• Data Warehouse Import: Best for importing directly from cloud warehouses like Snowflake, BigQuery, Amazon Redshift, Databricks, and Microsoft Fabric. Supports scheduled and one-time imports. For more information, refer to Data Warehouse Import.
• CSV Upload: Best for quick, one-time profile uploads via the dashboard (profiles only, up to 1 GB per file). For more information, refer to CSV Upload.
• Upload User Profiles API / Upload Events API: Best for programmatic, real-time or near-real-time data ingestion. For more information, refer to Upload User Profiles API and Upload Events API.

Do I need to upload user profiles before events?

Yes, it is strongly recommended to upload user profiles before events. This ensures that:

• CleverTap can correctly stitch events to the right user profiles during ingestion via identity resolution.
• Events uploaded without a matching profile may result in incomplete or anonymous user records, affecting segmentation and campaign targeting accuracy.
• Lifecycle metrics (for example, First Event, Last Event) are computed accurately.

For more information, refer to Upload User Profiles and How Profiles Merge.

What information should I include in user profile uploads?

To enable effective personalisation and targeting, include as much user information as possible. At a minimum, every profile upload must include a valid User Identifier (Identity). Recommended fields include:

• Mandatory: Identity (unique user ID such as internal DB ID, CRM ID, UUID, or any unique string).
• System User Properties: Name, Age, Email ID, Phone number (in +[country code][number] format). For more information, refer to System User Properties.
• Custom Profile Attributes: Any business-specific properties (for example, membership tier, preferred language).
• Nested Objects: User properties with nested structures for complex data. For more information, refer to Nested Objects in User Properties.
• Communication Preferences: Opt-in/opt-out status for channels such as Email, SMS, WhatsApp, and Push.

Ensure all date properties use Unix epoch format (seconds). For more information, refer to Schema for supported data types.

Can I test historical tested before sending them to production?

Yes, CleverTap strongly recommends testing with a sample dataset in your test account before importing into production.

The following are the key points to remember before sending them to production:

• Your test account can be reset; production cannot delete or modify imported data.
• The test account has a default 1-month Data Retention Policy (DRP), so ensure your sample data falls within this window.
• Validate correct identity stitching, event-to-profile mapping, timestamp accuracy, data type consistency, and field mapping in the manifest file during testing.
• For API-based imports, use the dryRun parameter (?dryRun=1) to validate payloads without ingesting data.

How does CleverTap handle timestamps on historical events?

When uploading historical events, include the original event occurrence time in the payload:

• For API, timestamps must be in Unix epoch format (seconds).
• For SFTP and Data Warehouse imports, CleverTap supports multiple date-time formats, as configured in the manifest file or during mapping (e.g., dd/MM/yyyy HH:mm:ss, yyyy-MM-dd, epoch). For more information, refer to SFTP Import – Date Time Formats and Data Warehouse imports - Date Time
• Events appear in dashboards and analytics based on the provided event time, not ingestion time.
• If no timestamp is provided, CleverTap records the event at ingestion time, potentially leading to inaccurate historical reporting.
• Import eligibility depends on your account’s Data Retention Policy (DRP). Events outside the DRP window may not be available for analysis.
• For the Test Account, the default DRP is 1 month.

📘

Note

All timestamps are treated as UTC (Coordinated Universal Time) unless otherwise specified in the manifest configuration.

How can I delete a Data Warehouse connection that has a running import?

To delete a Data Warehouse connection with active imports:

• Navigate to Settings > Partners > Import > Import Connections.
• Select the connection you want to delete and click Delete.
• Review any active imports. If imports are in Ready, Scheduled, Paused, or Running states, you must stop all of them before proceeding.
• Once all imports are stopped, click Delete again and confirm your action.

How can I whitelist IPs for CleverTap integration?

To ensure seamless communication between CleverTap and your data warehouses, whitelist the following required IP ranges based on your CleverTap dashboard region:

What are the recommendations while uploading historical data to CleverTap?

To ensure accurate ingestion and reporting, follow these best practices:

• Upload user profiles before uploading events
• Maintain consistent User Identifiers across profiles and events
• Upload events in chronological order (oldest first)
• Validate a sample dataset in your test account before production import
• Ensure timestamps are sent in Unix epoch format (seconds)
• Confirm that imported data falls within your account’s Data Retention Policy
• Break large datasets into smaller batches where applicable
• Ensure column headers are present in all CSV files (mandatory for SFTP import).
• Use consistent data types for the same property across all uploads (for example, do not mix string and numeric values for the same attribute).
• Avoid using special or reserved values as Identity (e.g., null, undefined, none, blank strings, or test). For more information, refer to CSV Upload – FAQs
• For SFTP imports, every data file must be accompanied by a manifest file. For more information, refer to SFTP Import

Following these practices helps ensure proper identity stitching and accurate lifecycle computation.

For data like date of birth, which date format does CleverTap support?

The supported date format depends on the import method:

• API: Date properties must be sent in Unix epoch format (seconds). CleverTap converts the timestamp internally into a date value for segmentation and reporting.
• SFTP imports and Data Warehouse imports: Multiple date-time formats are supported as configured in the manifest file (for example, dd/MM/yyyy, yyyy-MM-dd'T'HH:mm:ss, epoch) or while configuring mapping. If the timestamp is missing from a date-time field, CleverTap adds a default timestamp of 00:00:01.
• CSV Upload (Dashboard): Supports the $D_ prefix notation for epoch timestamps (for example, $D_1695801332). The data type must be set to alphanumeric in the mapping. CSV upload also supports other data types based on the mapping.

For more information, refer to SFTP Import – Date Time Formats, Data Warehouse imports - Date Time, and CSV Upload

How can I define my User Identifier in CleverTap?

The User Identifier (Identity) is a unique value used to distinguish one user profile from another.

You can define Identity using:

• Internal database user ID
• CRM ID
• UUID
• Any unique string value

The Identity value must:

• Be unique per user
• Remain consistent across profile and event uploads
• Not be reassigned to a different user
• Identity value must be less than 1024 characters.
• In addition to Identity, CleverTap also supports objectId [CleverTapID] (a system-assigned identifier). At least one must be present.
• For SFTP imports, the objectId column header is mandatory.
• For the CSV Upload (Dashboard), the identity (lowercase i) column is mandatory.

For more information, refer to User Profiles and How Profiles Merge.

What are the limitations while importing data to CleverTap?

When importing historical data, consider the following:

• Data imported into production cannot be rolled back
• Events outside your Data Retention Policy window will not be available for analysis
• Duplicate event handling must be managed externally
• Timestamps must be provided in Unix epoch format (seconds)
• SFTP supports files up to 5 GB per file. CSV Upload supports files up to 1 GB per file. Encrypted files (PGP) must be under 1 GB; split larger files before uploading.
• SFTP import and Data Warehouse import currently do not support importing the items array for the Charged event.
• Event names in CleverTap are case-sensitive (for example, Product Viewed and product viewed are treated as separate events).
• Upload APIs have a limit of 1,000 objects per 4 MB per request, with approximately 30 requests per second throttle.
• All files must be in CSV format and accompanied by a manifest file.
• Map and Array data types are currently not supported in dashboard CSV uploads and Data Warehouse Imports.
• It is recommended to validate all imports in a test account before proceeding to production.

Which file formats does import support?

The supported file format depends on the import method:

• SFTP Import: CSV format only. Each CSV file must be accompanied by a manifest file (JSON) that maps your columns to CleverTap properties. For more information, refer to SFTP Import
• SFTP with Encryption (PGP): CSV.ASC File format. Encrypted files must be under 1 GB. For more information, refer to File Upload Encryption
• Data Warehouse Import: Data is read directly from your warehouse tables/views (Snowflake, BigQuery, Redshift, Databricks, Microsoft Fabric). No file format conversion needed. For more information, refer to Data Warehouse Import
• CSV Upload (Dashboard): CSV format only, for user profiles. For more information, refer to CSV Upload
• APIs: JSON format. For more information, refer to Upload User Profiles API and Upload Events API

What is a manifest file, and is it mandatory for SFTP imports?

Yes. A manifest file is mandatory for every SFTP import. It is a JSON file that maps the columns in your CSV file to CleverTap properties. The format is anyfilename.manifest.

The following are the key points to remember:

• Every manifest file must have a unique name.
• Data file processing starts only after the corresponding manifest file is uploaded.
• The manifest file specifies the data type, CleverTap property name, and date format (if applicable) for each column.
• Uploading a CSV without its manifest file will result in an error.

For sample manifest files, refer to SFTP Import – Sample Manifest Files.

How can the status of my imports be monitored?

CleverTap provides multiple ways to track import status:

• SFTP: Go to Settings > Import > Activity Log to view uploaded files and their status. You will also receive email notifications at key milestones such as file pickup, processing, success, or failure. You can also configure webhooks for notifications.
• Data Warehouse: Go to Settings > Partners > Import > Import Connections to view import run history and stats. For more information, refer to Import Listing and Stats. You will also receive email and CleverTap Dashboard notifications at key milestones such as Import Start, Import Completed with success, or Import Completed with failure.
• CSV Upload: Go to Settings > Engage > CSV Uploads. The status for CSV upload is shown as Completed, Processing, Rejected, and Start Mapping. You will also receive email notifications at key milestones such as file processing, success, or failure.

When an import fails, you receive a failure email with a CSV attachment containing details of the failed records and error reasons for SFTP and CSV upload.

How do I troubleshoot failed imports?

When an import fails, perform the following steps:

• Check email notifications: Failed imports trigger an email with a CSV attachment listing the failed records and error descriptions in the last column for SFTP and CSV upload. For Data Warehouse import, an email with the failed count and a link to the Stats page gets triggered. You can also go to Settings > Partners > Import > Import Connections to view detailed import stats. For more information, refer to Import Listing and Stats.
• Review the Activity Log: For SFTP, go to Settings > Import > Activity Log. For the Data Warehouse, check the Import Run Details under Import Connections.
• Common errors include: missing or invalid Identity/objectId, incorrect data types, malformed manifest file, timestamps outside the DRP window, file format issues, and column header mismatches. For the data warehouse, check permissions/access.
• Error Stream: Use the Error Stream feature (accessible from the main menu) to view ingestion errors over a specified time period.
• Validate first: Always test a sample dataset in your test account before running a full production import.

Can I schedule recurring imports from a Data Warehouse?

Yes. CleverTap supports both one-time and recurring scheduled imports from your data warehouse. You can configure the import frequency (for example, daily, weekly, or custom intervals) during the scheduling step. For more information, refer to Schedule Import.

For SFTP, you can also automate recurring imports by writing a script that connects to CleverTap SFTP and transfers data at defined intervals.

Can I import encrypted files to CleverTap?

Yes. CleverTap supports PGP encryption for secure SFTP file uploads. With File Upload Encryption, your data files are encrypted before transfer and decrypted securely within CleverTap’s environment.

The following key points to remember:

• Supports CSV.ASC file formats.
• Encrypted files must be under 1 GB. Split larger files before uploading.
• You can generate PGP key pairs either self-generated or via CleverTap.
• Each encrypted file must be accompanied by a manifest file that includes the Key ID.

For more information, refer to File Upload Encryption.

How does CleverTap handle duplicate events during import?

CleverTap does not perform automatic deduplication for imported events. If the same event with the same timestamp and properties is uploaded multiple times, it will be recorded as separate event entries. You must manage duplicate detection and handling externally before importing.

For user profiles, if an existing profile with the same Identity is uploaded again, the old profile is updated with the new values (upsert behavior).

What happens if I upload events for users that do not exist in CleverTap?

If events are uploaded for users whose profiles do not yet exist in CleverTap, the system creates an anonymous user or one with the provided identity. This can lead to fragmented user data and inaccurate lifecycle computation. To avoid this, always upload user profiles before uploading events.

For more information on identity resolution and profile merging, refer to How Profiles Merge.

Can I import event data via CSV upload?

No. Currently, the CSV Upload feature (Settings > Engage > CSV Uploads) supports user profiles only. To import events, use SFTP Import, Data Warehouse Import, or the Upload Events API.

What happens if I upload data with an incorrect data type for an attribute?

CleverTap validates data types during ingestion. If a value does not match the expected data type for a property (as defined in the schema or manifest file), the record may fail and appear in the error report.

To avoid this failure:

• Ensure numeric fields contain numbers, date fields contain valid dates, and string fields do not exceed the character limit.
• Use the manifest file (for SFTP) or column mapping (for CSV Upload and Data Warehouse Import) to explicitly define data types for each field.
• Review your property definitions under Schema to confirm expected data types before importing.

What character encoding should my import files use?

All CSV files uploaded to CleverTap via SFTP, CSV Upload, or encrypted file upload must use UTF-8 encoding. Files saved in other encodings, such as Latin-1 or Windows-1252, may cause data corruption, especially with non-English characters.

📘

Best Practice

When saving CSV files from Excel, select CSV UTF-8 (Comma delimited) as the file format.

What are the rate limits or batch size restrictions for API-based imports?

Yes. When using the Upload User Profiles API or Upload Events API:

• Maximum of 1,000 objects per request with a 4 MB payload limit.
• Use the dryRun parameter (?dryRun=1) to validate payloads without ingesting data.

For SFTP imports, files up to 5 GB are supported. For CSV upload via the dashboard, the maximum file size is 1 GB.

How do I troubleshoot a Data Warehouse import connection failure?

If your Data Warehouse connection or test connection fails, check the following:

• IP Whitelisting: Ensure CleverTap’s IP ranges are allowlisted in your warehouse’s firewall/network rules.
• Credentials: Verify that the service account, username, password, or key file used in the connection setup has the correct read permissions on the source table/view.
• Table/View access: Confirm that the table or view referenced in your import configuration exists and is accessible by the configured service account.
• Network/VPN: If your warehouse is behind a VPN or private network, ensure the necessary routing is in place.

For detailed setup instructions for each warehouse, refer to Set Up Import.

Are there any naming conventions or limits for custom attributes?

Yes. When importing data with custom user properties or event properties, ensure the following points are taken into consideration:

• Property names should be descriptive and consistent across all import methods (API, SFTP, Data Warehouse).
• Avoid using reserved system property names as custom property names.
• Once a property name and data type are set in CleverTap, they cannot be changed. Plan your schema carefully before the first import.
• Event names are case-sensitive.

For best practices on naming conventions of user and event properties, refer to Schema.