Overview

IntelliAB is CleverTap's AI-powered experimentation layer built on top of A/B testing. It enables marketers to test up to 10 message variants simultaneously within a Push campaign and uses a Multi-Armed Bandit (MAB) algorithm to continuously learn which variants drive the most engagement, automatically shifting traffic toward better-performing variants in real time.

Manual A/B testing works well with a small number of variants but becomes inefficient as experimentation scales. When too many variants split your traffic equally, most users see weaker messages, and results only tell you who won and not why a particular variant won. IntelliAB Test solves this by turning experimentation into a continuous, self-optimizing system.

For example, you are running a flash sale. You create 4 push notification variants with different CTAs, tones, and offer framing.

With a Manual A/B Test, you split traffic evenly (25% each) and wait 7 days to determine a winner. By the time you act on results, the sale is over. Users who saw the weaker variants never converted.

With IntelliAB, by day 2, the bandit has already identified that Variant C is getting 3x the clicks and automatically routes 60% of traffic there, while the sale is still live. And it never stops learning. If Variant D starts outperforming C on day 5, traffic shifts again, automatically.

Manual A/B Test vs. IntelliAB Test

The following table shows how the IntelliAB Test differs from the manual A/B Test:

Key Concepts

• Exploration: A configurable percentage of your audience that always receives all variants equally, regardless of which variant is currently winning. This ensures that every variant continues to accumulate data, preventing the system from prematurely locking onto an early leader.
• Exploitation: The remaining traffic is dynamically allocated to the variants the AI model currently believes perform best.
• Arms: Each message variant is an "arm" of the bandit. CleverTap supports up to 10 arms (variants) per campaign.
• Reward Signal: The event you define as success. When a user achieves this event after receiving a variant (for example, clicking the notification), that variant receives a "reward" that improves its standing in the model.

IntelliAB Test Workflow

Algorithm

IntelliAB Test uses a goal-based MAB algorithm to dynamically allocate traffic across variants. For each variant, CleverTap maintains two counters:

• Total messages sent
• Total goal events achieved (rewards)

These feed a statistical model (Beta distribution) per variant. As more data accumulates, each variant's model becomes more precise, allowing the algorithm to distinguish genuine winners from early flukes.

The algorithm balances two competing objectives simultaneously:

• Exploration: A configured percentage of traffic is always distributed equally across all variants, regardless of which is currently winning. This ensures that every variant continues to receive data throughout the campaign.
• Exploitation: The remaining traffic is allocated by the AI model, which routes users toward the variant most likely to achieve the configured goal based on current performance data.

The degree of exploration vs. exploitation is controlled by the Exploration % setting configured before publishing the campaign. A higher exploration % means traffic is more evenly distributed across all variants; a lower % means a larger share of traffic is routed by the AI model toward the leading variant.

Traffic Allocation

For every user who qualifies for the campaign, the system makes a real-time routing decision:

• Exploration bucket (your configured Exploration %)

  The user is assigned uniformly at random to any variant, regardless of current performance. This is the always-testing portion that keeps all variants alive.

• Exploitation bucket (remaining % of traffic) The system samples from each variant's probability model and routes the user to whichever variant it predicts is best at that moment. Variants with consistently higher goal rates get selected here more often.

What Counts as Reward?

The reward signal is the goal metric that the marketer selects when configuring the experiment. Notification Clicked is the supported goal metric.

Every time a user who received a specific variant triggers your chosen goal event, the variant reward count increases. This improves the model score and share of exploitation traffic.

How Variant Weights Update Over Time?

After each campaign run (recurring) or on an ongoing basis (live/trigger), the system:

1. Reads the latest Sent + Goal-Met counts for all variants.
2. Rebuilds each variant's probability model from the updated data.
3. Uses the updated model for all subsequent user routing decisions.

Better-performing variants (higher goal rate per send) progressively capture a larger share of the exploitation traffic. The gap widens as data accumulates.

When Does IntelliAB Test Declare Winner?

The campaign runs continuously and optimizes in real time; there is no fixed time window or sample size that triggers an automatic stop.

You declare a winner manually from the stats page when you are satisfied. CleverTap supports this decision by indicating the variant with a Statistically significant badge when its probability of being the best performer is 90% or higher. This indicates that the data is reliable enough to act on.

Once you declare a winner, 100% of traffic goes to that variant for the remainder of the campaign.

Set Up IntelliAB Test Campaign

The setup involved the following six significant steps:

1. Create A/B Test Push Campaign.
2. Define Variants.
3. Configure Experiment Mode and Goal.
4. Set Exploration Rate.
5. Configure Delivery Preferences.
6. Publish Campaign.

Create A/B Test Push Campaign

To create a Push campaign that uses an IntelliAB test mechanism, perform the following steps:

1. Go to Campaigns and click + Campaign.
2. In the What step, select A/B Test as your message type.

Define Variants

You can create your campaign message manually or use the Fill with AI option. AI fills the title and message of the current variant. For more information about creating a message using AI, refer to Scribe.

To create variants, you can do it manually by clicking or by clicking Create More Variants. AI generates a new variant based on the intent of your existing copy. All non-copy fields are auto-cloned from the source.

Each variant supports Clear, Delete, and Copy controls, the same behavior as standard A/B. Edits to each variant are independent and persist separately.

Configure Experiment Mode and Goal

Once the variants are set, save them, then click Done. The campaign Overview page opens. To configure the Experiment Mode and Goal, perform the following steps:

1. Set the User distribution to Automated.
2. Under Win criteria, select your goal event, that is, Notification Clicked

The Win criteria event is used to compare variant performance and determine the winner.

Set Exploration Size

Exploration Size is the percentage of your audience that will always receive all variants equally, even as one variant starts winning.

Percentage of users who receive all variants, split evenly, even after a variant is winning. In the first run, variants are split evenly across the segment.

Configure Delivery Preferences

Define the campaign schedule. An IntelliAB Test campaign is available only for campaigns that run more than once:

If you switch delivery from recurring/trigger to one-time after configuring Automated mode, the experiment settings reset to Manual A/B Test.

Publish Campaign

Review all variants, your goal metric, exploration %, and delivery type, and publish the campaign when ready.

The first run distributes traffic evenly across all variants, regardless of your exploration setting, seeding the learning model. Optimization begins from the second run onward.

IntelliAB Test Campaign Stats

You can view the campaign stats by selecting the campaign on the campaigns list page, then selecting the Stats tab. For IntelliAB Test campaigns, the stats page includes the following:

• Variant-Level Performance Table
• Goal Conversion Trend Chart
• Statistically Significance
• Declare Winner

Variant-Level Performance Table

The campaign Stats page displays a variant-level performance table. The table includes:

An All Variants aggregate row is shown at the top of the table. CSV download is supported for all variant data.

Goal Conversion Trend Chart

A line chart shows each variant's goal conversion % over time. This visualizes the learning progression; you can see how the IntelliAB Test identifies and converges on the better-performing variants.

Time granularity adjusts based on campaign type: Hourly for Live campaigns, Daily/Weekly/Monthly for PBS campaigns.

Statistical Significance

When a variant reaches 90% or more probability of being the best performer, CleverTap highlights it with a Statistically Significant badge, shown as a green checkmark next to the variant in the stats table.

Declare Winner

IntelliAB Test campaigns have no fixed test window; the experiment runs continuously for the campaign lifecycle. You do not need to stop the campaign to get value from the IntelliAB Test. However, you may want to manually declare a winner when:

• One variant's goal conversion rate is clearly and consistently higher than that of other variants.
• The campaign has run long enough for the distribution trend chart to show stable allocation.
• You want to lock in a variant and stop further exploration.

Best Practices

Wait for the statistical significance badge before acting

An IntelliAB Test has no fixed sample size requirement; it runs continuously. But results early in the campaign (especially the first run) are unreliable, as all traffic is distributed equally to seed the model. The statistical significance badge is your reliable signal to act.

Ideal campaign types for IntelliAB Test

IntelliAB Test compounds its advantage across multiple runs. It works best for:

• Recurring push campaigns (weekly/monthly re-engagement, promotional)
• Always-ON trigger campaigns (onboarding, lifecycle nudges)

Avoid using an IntelliAB Test for a campaign you will only send once. It will not have enough runs to optimize meaningfully.

Choosing Right Goal Event

You must pick based on your campaign objective:

Setting Exploration %

• Do not go too low: The system can get stuck on an early leader. If that variant's performance later drops (audience shift, seasonal change), you'll have too little data on others to pivot.
• Do not go too high: Exploitation never gets enough traffic to meaningfully reward the best variant.