May 22, 2025
A/B tests analytics
This article describes the metrics in detailed A/B test reports and how to find them.
Step-by-step instructions
Navigate to the "Outbound" section.
Click the "A/B tests" button.
Press the “Analytics” button.
Data filtering by specific periods: "During the test period" or "During the entire duration".
Exporting analytics for a specific A/B test in PDF or EXCEL format for further reporting.
-
Analytic parameters
- Channel & Status - Displays the channel type and current A/B test status (running, completed, or halted).
- Name - The designated name of the A/B test campaign.
- Test components - The variable being tested: Content, Subject line, From name, or Launch time.
- Template (A, B, C) - The names of the specific templates used for each test variant.
- Email Subject (A, B, C) - The different subject lines being compared (Email channel only).
- From Name (A, B, C) - The sender names are being tested to determine which drives the most engagement.
- Launch Time (A, B, C) - The specific dispatch times for each variant when testing the optimal send time.
- Criterion - The defined metric used to determine the winner (Open rate, Click rate, or Unsubscribe rate).
- Duration - The time frame allocated for the test (from 30 minutes up to 24 hours) before a winner is chosen.
- Winner Variant - Displays the winning version once determined, or statuses like "No winner yet" if the test is ongoing or halted.
- Winning message send time - The scheduled date and time for sending the winning variant to the remaining audience.
- Chart Overview - A visual hourly dynamic of the selected criterion, showing how each variant performs over time (available for tests longer than 3 hours).
- Statistic Type - A toggle between "During the test period" (data only from the test phase) and "During the entire duration" (cumulative data including the final dispatch).
- Variant List - A breakdown of all tested versions and the final "Winner" message (marked with an icon).
- Recipients - The number of contacts who received the individual test variants and the final winning message.
- Delivered - Status assigned once the message is successfully accepted by the recipient's provider server.
- Open Rate / Opens - The percentage of unique opens (Email, Telegram, Viber) or viewers (WebPush) for each variant.
- Click Rate - The percentage of unique recipients who clicked a link within a specific test variant.
- Email | Unsubscribed - The percentage of recipients who opted out after receiving a specific test version.
- Email | Hard Bounce - Permanent delivery failures; these addresses are automatically blacklisted.
- Email | Soft Bounce - Temporary delivery failures (e.g., mailbox full); the system retries for 3 days.
- Email | Marked as Spam - The count of recipients who manually reported the A/B test email as spam.
- Email, Viber | Rejected - Messages not sent due to blacklist status, daily limits, or anti-spam triggers.
- WebPush, Viber | Failed - Instances where messages couldn't be delivered due to blocked accounts or unavailable recipients.
- WebPush, Viber | Expired - Notifications or messages not shown because the TTL (Time to Live) duration expired.
- Excel (XLSX) Export - Downloads all result data in a tabular format, optimized for Google Sheets.
- PDF Export - Downloads a visual report including charts and graphs exactly as they appear on the dashboard.
