Overview

CleverTap’s Message Archiving feature stores a copy of each outgoing message sent through campaigns or journeys to your AWS S3 bucket—such as a push notification, email, SMS, or WhatsApp message. By storing copies of outgoing messages from campaigns and journeys across multiple channels in real-time and provides evidence of compliance. It allows access to archived communication for compliance, audit, or customer support.

📘

Enable Message Archiving

The feature is available as a paid add-on. To enable this feature, contact your Success Manager or the CleverTap Support..

Supported Channels

Currently, Message Archiving is available for the following channels:

• Email
• Push notifications
• SMS
• WhatsApp

How It Works

When the Message Archiving feature is enabled, CleverTap will:

1. Archive outgoing messages to your AWS S3 bucket in real-time.
2. Send messages through selected channels as part of campaigns and journeys.

🚧

Message Delivery Lag

Enabling this feature may introduce additional latency, as the archiving is performed before message delivery to ensure accuracy.

File Structure

Check that the file structure and path follow the guidelines. The archived message files on the AWS contain a JSON payload.

File Structure and Path

The archived messages are saved in your AWS S3 bucket using a specific key structure, helping organize and retrieve the files efficiently. The general structure of the file path is as follows:

sent_messages/<channel>/<yyyymm>/<campaign_id>/<user_id>/message_id.json.lz4

where:
<*> = folder
<yyyymm> = year, month // batchId(<yyyymmdd> year, month, day) from the message
<channel> = Push OR Sms OR Email or WhatsApp 
<user_id> = MD5 encoded data of Device Identity if exists, else, Customer Identity for Push, Mobile Number in CountryCode format for Sms & Whatsapp, EmailId for Email channel
message_id.json.lz4 is a lz4 compressed file representing one message. // support lz4 compression

An example file path may look like this:

sent_messages/WhatsApp/202409/1725435360/6e6d748a7cc752fbe4f72c777cc50a74/49482d489b871510eeff7db9a8ab443248ad1ea596e5108e02d16ab5b2ea47db.json.lz4

Sample JSON Payloads

The archived files on the AWS contain a JSON payload representing the final message sent to the user. Here is an example of the JSON structure:

{
  "version" : 1 //numerical version of the json structure,
  "to": "frfsgvrwe:APfdsafsgdfsgghfgfgjkhfsfgdhjhbvcvnetry767456fxsasdf"
     // platformId or pushToken,
  "payload": JsonOfEntirePushPayload,
  "platform": "iOS" or "Android",
  "sent_at": 1722443282680 // Message Archiving epoch in millis,
  "message_id": "57040f2b01ac851c2b2538e817983d1e2e6cdd01cc5e20740550729525a3f345"
         // sha256 hash of accountId-targetId-batchId-platformId-MessageCreationTs-userId,
  "campaign_id": 1722529032 // the campaign ID is unique to each campaign, will always exist,
  "campaign_name": "Flat Offer Sale Campaign" // the campaign Name for current campaign, will always exist,
  "user_id": 45142169 // Clevertap internal Device Identity if exists, else, Customer Identity, 
  "journey_id": 2450 // may not be available, exists only if this campaign is a journey node,
  "journey_node_id": 1 // may not be available, exists only if campaign is a journey node,
  "journey_name": "Flat Offer Sale Journey", // may not be available, exists only if campaign is a journey node
}

{
  "version" : 1, //numerical version of the json structure
  "to": ToAddress, ("[email protected]")
  "subject": SubjectLine ("20% off coupon inside!"),
  "from_name": DisplayName ("CleverTap"),
  "from_address": FromAddress ("[email protected]"),
  "cc": // List of CC's if available, currently in beta mode per email team,
  "bcc": // List of BCC's if available, currently in beta mode per email team,
  "html_body": HtmlBody,
  "plaintext_body": PlainTextBody,
  "amp_body": AMPEmailBody, // will only be available if the email has ampBody
  "headers": headers for the email,
  "sent_at": 1722443282680 // Message Archiving epoch in millis,
  "message_id": "5fae1e791db5f2e61ef39fb34a853aa6e9c8847e78e2766c888d10ee65221abc" 
     // hash of accountId-campaignId-userId-toAddress-batchId-MessageCreationTs,
  "campaign_id": 1722529032 // the campaign ID is unique to each campaign, will always exist,
  "campaign_name": "Flat Offer Sale Campaign" // the campaign Name for current campaign, will always exist,
  "user_id": 45142169 // Clevertap internal Device Identity if exists, else, Customer Identity, 
  "journey_id": 2450 // may not be available, exists only if this campaign is a journey node,
  "journey_node_id": 1 // may not be available, exists only if campaign is a journey node,
  "journey_name": "Flat Offer Sale Journey", // may not be available, exists only if campaign is a journey node
}

{
  "version" : 1 //numerical version of the json structure
  "to": PhoneNumber, ("15555555555"),
  "body": Body ("Hi there!"),
  "provider": "twilio" // StringOfProviderName,
  "sent_at": 1722443282680 // Message Archiving epoch in millis,
  "message_id": "9f8c76ccdaf0895515f813af702670324d996296788652b50b3475d6fe6e719a" 
        // hash of accountId-userId-batchId-campaignId-toPhoneNumber-MessageCreationTs,
  "campaign_id": 1722529032 // the campaign ID is unique to each campaign, will always exist,
  "campaign_name": "Flat Offer Sale Campaign" // the campaign Name for current campaign, will always exist,
  "user_id": 45142169 // Clevertap internal Device Identity if exists, else, Customer Identity, 
  "journey_id": 2450 // may not be available, exists only if this campaign is a journey node,
  "journey_node_id": 1 // may not be available, exists only if campaign is a journey node,
  "journey_name": "Flat Offer Sale Journey", // may not be available, exists only if campaign is a journey node
}

{
  "version" : 1 //numerical version of the json structure
  "to": PhoneNumber, ("15555555555"),
  "body": Body ("Hi there!"),
  "provider": "twilio" // StringOfProviderName,
  "sent_at": 1722443282680 // Message Archiving epoch in millis,
  "message_id": "9f8c76ccdaf0895515f813af702670324d996296788652b50b3475d6fe6e719a" 
        // hash of accountId-userId-batchId-campaignId-toPhoneNumber-MessageCreationTs,
  "campaign_id": 1722529032 // the campaign ID is unique to each campaign, will always exist,
  "campaign_name": "Flat Offer Sale Campaign" // the campaign Name for current campaign, will always exist,
  "user_id": 45142169 // Clevertap internal Device Identity if exists, else, Customer Identity, 
  "journey_id": 2450 // may not be available, exists only if this campaign is a journey node,
  "journey_node_id": 1 // may not be available, exists only if campaign is a journey node,
  "journey_name": "Flat Offer Sale Journey", // may not be available, exists only if campaign is a journey node
}

Browse Archived Message From AWS S3 Bucket

The following image shows the steps to browse the archived messages on the AWS S3 bucket:

Set Up Message Archiving

Prerequisites

Before setting up Message Archiving, check that you have:

• An active CleverTap account.
• An AWS S3 bucket in the same region as your CleverTap account region.

Set Up Message Archiving

The process involves the following two steps:

1. Configure AWS S3 Bucket on CleverTap dashboard: Provide the necessary details for your AWS S3 bucket, including the bucket name and region. For more information, refer to Configure AWS S3 Bucket on CleverTap Dashboard.
2. Configure Data Retention Policy: You can set up the data retention policy for your AWS S3 bucket according to your organizational needs directly in the AWS S3 console.

For more information and detailed configuration steps, refer to Data Export to AWS S3.

Key Benefits

Some of the key benefits of archiving your messages are:

• Audit Readiness Maintains a comprehensive record of all user communications, ensuring adherence to regulatory compliance.
• Real-Time Archiving: Archives messages in real-time with minimal impact on daily operations.
• Low Maintenance: Requires little ongoing maintenance, freeing up resources for other tasks.

Security and Data Protection

CleverTap takes security seriously. Here's how your data is protected:

• Data in Transit: All data is transmitted using the HTTPS protocol, ensuring secure data transfer between CleverTap and your AWS S3 bucket.
• Data at Rest: The security of data at rest within the AWS S3 bucket is the customer's responsibility. This includes ensuring that proper encryption and access controls are in place.

Frequently Asked Questions (FAQs)

Explore concise troubleshooting tips and FAQs for Messaging Archival in this section.

Q: Is there a user interface for Message Archiving?

A: No, the Message Archiving feature operates without a user interface. Once enabled, it seamlessly archives messages to your AWS S3 bucket. A user interface will be available in the future.We will provide a user interface in the future.

Q: How long is the data stored?

A: You define the data retention policy within your AWS S3 bucket settings.

Q: Who bears the cost of storage?

A: The customer bears all storage and data transfer costs associated with their AWS S3 bucket.

Q: Does archiving happen in real-time?

A: Yes, message archiving occurs in real-time, ensuring that all communications are promptly saved.

Q: What happens if my AWS credentials are invalid?

A: If your AWS credentials become invalid, CleverTap cannot archive messages to your S3 bucket, and no further campaign messages will be sent to your users. This is to ensure that every message that is sent out is archived first.

Q: How are retries handled?

A: If the bucket is unreachable, we will retry up to three times for AWS errors (such as AccessDenied). AWS S3 rate limit errors are handled by CleverTap.

Q: What happens if archiving fails ?

If the archiving fails after multiple retries, messages are marked as Failed to Archive Message .

Q: Why does my archive sent_at timestamp differ slightly from the actual send time?

A: The message is archived just before it is sent to the end user, due to which slight delays can occur. This may lead to minor differences in timestamps.