A/B tests
This article describes how to create an a/b test using various advanced options.
Things to know before starting
- A/B tests can be sent to the entire Contact List or to a specifically defined Audience group.
- A group of 5,000 contacts is recommended for a Test group to ensure reliable results.
- Samples may not be representative if the contact group is too small; increasing the number of contacts is required to reduce the margin of error.
- The A/B Test Target group is dynamic, meaning the final list is determined when a variant or winner is launched.
- Messages are not sent to contacts who have opted out, are blacklisted, or no longer meet segment criteria at launch.
- Up to three variations of Email, WebPush, Telegram, or Viber messages can be sent to a small sample of the audience.
- Testing only one component at a time is advised to accurately identify which specific element has the greatest impact on open and click-through rates.
- The maximum length for the A/B Test Title is 500 characters.
- Data recalculation is triggered whenever a move from Step 1 to Step 2 occurs or when the recalculation button in the info table is pressed.
Step-by-step instructions
Navigate to the "Outbound" section.
Click the "A/B tests" button.
Click the "Create" button.
Entry of the A/B Test title in the designated field.
Select the Contact List for the testing process.
Configuration of a Segment to target a specific group of recipients (optional).
Use the Add filter option to create one-time segmentation rules that apply only to the current test and are not saved in the general segment list.
Application of the Exclude Segments feature to remove specific contact groups from the mailing list (optional).
Select the communication channel from the dropdown menu: Email, WebPush, Telegram, or Viber.
Use the preview pane on the right to see how variants appear across devices and operating systems.
Review of sender details (From, Reply to) and Subject lines for Email, or browser-specific rendering for WebPush.
Click the Next step button to proceed to the second configuration stage (the test assumes Draft status).
Setting the percentage of subscribers for the test group using the slider or Edit button (the contacts are divided equally among test variants).
Definition of the winning criteria: unique opens (Open rate), unique clicks (Click rate), or quantity of unsubscribes (Unsubscribe rate - Email only).
Select the A/B Test duration to set the data collection timeframe (30 minutes to 24 hours).
Configuration of winning variant actions: automated system sending at a scheduled time or manual selection based on reporting data.
Choice of the launch type: Manual (launch from the management page), Now (immediate sending), or Scheduled (postponed to a specific date and time).
Final review of all campaign details in the summary table before activation.
Click the “Launch A/B test” button.
-
Summary section parameters
- Contacts total - the total number of contacts identified based on the Step 1 conditions, representing the Target group for the test.
- Test group - the number of contacts participating in the test, displayed in both percentage and numerical formats.
- Variant A / B / C - the number of contacts assigned to each test variant group, presented in both percentage and numerical values.
- Winning message - the number of contacts who will receive the winning variant after the test is completed, shown in both percentage and numerical formats.
-
Details
- Contacts total (All channels) - the total number of contacts in the target group based on specified conditions.
- Test group (All channels) - the number and percentage of contacts participating in the testing phase.
- Variant A / B / C (All channels) - the numerical and percentage distribution of contacts within each test variant group.
- Winning message (All channels) - the number and percentage of contacts who will receive the winner after the test completes.
- Channel (All channels) - the communication channel type used (Email, WebPush, Telegram, or Viber).
- Name (All channels) - the internal title of the A/B Test campaign.
- Template A / B / C (All channels) - names of the templates being tested; clicking the name redirects to the template editor.
- Subject A / B / C (Email) - the subject lines undergoing testing for the Email channel.
- From name A / B / C (Email) - the sender name displayed to the client; can be a fixed value or a tested component.
- From email / Reply-to email (Email) - the email addresses used for sending messages and receiving customer replies.
- TTL / Time to live (WebPush, Viber) - the timeframe for delivery attempts before a notification is discarded.
- Telegram bot (Telegram) - the name of the selected bot used for delivery in the Telegram channel.
- Sender Name (Viber) - the identified name seen by contacts receiving a message via Viber.
- Criterion (All channels) - the specific metric used to define the winning message (Open rate, Click rate, etc.).
- Duration (All channels) - the time period allowed for gathering data before a winner is determined.
- Winning message send time (All channels) - the scheduled date and time for sending the winning variant to the remaining audience.
- Audience data (All channels) - details regarding selected contact lists, segments, and active exclusion filters.
- Target contacts count (All channels) - the final number of contacts in the campaign, with an option to refresh/recalculate.
- Launch type (All channels) - the selected execution method: Now, Scheduled, or Manual.
- Launch time A / B / C (All channels) - the overall start time or variant-specific start times for "Launch time" testing.
- Tags (All channels) - the list of optional labels used for organizing and managing campaign analytics.
