May 12, 2023

Manage A/B Tests

Email marketing is an essential component of any successful digital marketing Campaign. However, not all messages are created equal. To optimize your Campaigns, A/B Testing is a must.

In this article, we will show you all the controls on the A/B Tests page.

To start work with A/B Tests – follow the steps below:

Navigate to the A/B Tests Management page (Menu->Outbound->A/B tests).
Press the filter button to select the channel.
- All
- Email
- WebPush
- Telegram
- Viber
Press the filter button to select the status.
- All
- Draft
- Waiting
- Ready
- Running
- Done
- Stopped
- Failed
Sort A/B Tests by name.
Sort A/B Tests by test component.
Search A/B Test by name and by ID.
Press the “Analytics” button. The report shows the performance of your A/B Test Campaign and its key metrics.
Press the “Info” button to get acquainted with the A/B Test details.
By pressing the “More” button the following actions will be available:
- Duplicate – create the new A/B Test with the same settings and parameters.
- Delete – remove the A/B Test without a recovery option. NOTE: The A/B Test could be also deleted by choosing the selection box next to the A/B Test row.
Edit the desired A/B Test by pressing on its name.

The interaction of buttons and statuses

Button	Status	Launch type
Ready	(Ready for sending, can be launched, and can be edited before launch.)	Manual
Running (The stop/disable button)	(Cannot be stopped and edited.)	Now Manual Schedule
Waiting	(If you pause an A/B Test and try to restart it after the launch time has passed, an alert about the missed time will be displayed. You can still edit such a Campaign.)	Schedule
Stopped	(If the A/B Test was stopped before sending. Can be edited and launched.)	Schedule
Done	(Cannot be edited and cannot be launched.)	Now Manual Schedule
Draft The A/B Test is not active	(If you leave the A/B Test setting page after pressing the “Next step” button. Can be edited. Cannot be launched.)	Not selected
Failed	(The reason can be seen in the corresponding table by pressing the “Info” button. Can be edited.)	Now Manual Schedule

A/B Test Analytic parameters

Section	Parameter	Displays for	Description
The A/B Test detailed information	Channel & Status	All channels and test components.	Shows the A/B Test Campaign channel and its status at the moment of the Analytics checkup.
	Name	All channels and test components.	The name of the A/B Test Campaign.
	Test components	All channels and test components.	The tested key element of the Campaign (Content, Subject, From name, Launch time).
	Template	All channels and test components except for the “Content” component.	The Template that undergoes A/B Testing.
	Template A	All channels but “Content” test component only.	The names of the Templates that are undergoing A/B Testing.
	Template B
	Template C
	Email subject	Email Channel. For the “From name”, and “Launch time” components only.	The subject line of the A/B Test email Campaign.
	Email subject A	Email Channel. For the “Content” and “Email subject” components only.	The subject lines of the emails that are undergoing the A/B Testing.
	Email subject B
	Email subject C
	From name	Email channel. For all test components except the “From Name” component.	The provided name that is displayed in your recipient’s inbox.
	From name A	Email channel. For the “From name” component only.	The From names that are undergoing A/B Testing.
	From name B
	From name C
	From email	All components of the Email channel.	The email address from which the A/B Test Campaign is sent.
	Reply-to email	All components of the Email channel.	The email address to which your client can send the reply.
	Reply-to name	All components of the Email channel.	The name to which your client can apply for the reply.
	Sender Name	All components of the Viber channel.	The selected Sender Name that your contacts saw when they received a Viber message from your company through the Wooxt.com platform.
	Tags	All channels and test components.	The list of Tags of the A/B Test.
	TTL (Time to live)	All test components of the WebPush and Viber channels.	The defined duration of the push service attempts to deliver the notification or Viber service to deliver the message.
	Telegram bot	All test components of the Telegram channel.	The name of the selected Telegram bot.
	Contact list	All channels and test components.	The name of the selected Contact List.
	Segments	All channels and test components.	Information about the selected Segments that determined the A/B Test Target group.
	Exclude segments	All channels and test components.	Information about the filter, that excluded contacts from the Target group.
	Filtered by	All channels and test components.	Information about the temporarily created segment.
	Excluded by filter	All channels and test components.	Information about the one-time filter, that excluded contacts from the Target group.
	Launch time	All channels. For all test components except the “Launch time” component.	The A/B Test starting time.
	Launch time A	All channels. For the “Launch time” component only.	A/B Test start time for each variant.
	Launch time B
	Launch time C
	Criterion	All channels and test components.	Defined criteria for choosing the winning message.
	Duration	All channels and test components.	A/B Test Campaign duration. The winner is determined based on the data for the specified time frame.
	Winning message send time	All channels and test components.	The winner sending date and time. Displayed only for the “send the winning message automatically” option.
The export report button	Excel	All channels and test components.	Downloads all data from the Results table in tabular form (no graphs) in XLSX format on your device. `NOTE` The most comfortable way for the downloaded file to read is to open it via Google Sheets.
The export report button	PDF	All channels and test components.	Downloads data and graphs the way they appear on the page in PDF format.
Winner determination	Winner variant	All channels and test components.	Displays information if the winner is not determined yet: NO WINNER YET- the test is still running; NO WINNER YET – The process of sending to the winner has been halted. Displays the name of the winner – the variant that outperformed all other variations. When the “Content” component was tested, then the displayed Template name of the winner will be clickable.
	Criteria for choosing the winning message	All channels and test components.	Previously defined criteria for choosing the winning message: By open rate – based on unique opens. By click rate – based on unique clicks. By unsubscribe rate – based on the quantity of unsubscribed contacts after the test variants are sent.
	Test duration	All channels and test components.	The specified time frame, during which the winner is determined: 30 minutes 1 hour 3 hours 6 hours 12 hours 18 hours 24 hours
	Winning message send time	All channels and test components.	Appears after the winner is already determined. Displays the date and time when the winning variant will be sent to the rest of the Target group.
	Test components	All channels and test components.	The list of Test components that are objects of the A/B Testing: Content (the names of the Templates are clickable) Email subject From name Launch time
Chart overview	Test components and Final dispatch	All channels and test components. NOTE: Appears only if the duration of the Test is three or more hours. NOTE: Reflects hourly-based metrics dynamic, accordingly to the criterion selected earlier to determine the winner. Each bullet displays a data summary for the last hour.	Horizontal lines show you the dynamics of Test variants and the winner. By hovering the cursor on the bullets the following information is displayed: The Test time frame which has already passed. Test components. The number of contacts that have met the conditions of the winner determination as a result during the previous hour time frame. The Test group percentage for the particular Test variant. You also can view separately only one of the indicators – “Variant A”, “Variant B”, “Variant C”, or “Final dispatch”. Just click on the corresponding field to exclude the parameters view.
Statistic type	During the test period	Al channels and test components.	This option is selected by default. The table below will show you only the data related to the test period.
Statistic type	During the entire duration	All channels and test components.	Select this option from the drop-down menu if you want to display all the data from the very beginning till the moment you check the Analytics.
Results table Reflects all the parameters that occurred during the Test period or during the entire duration.	Variant	All channels and test components.	The list of the tested variants and the winning message. The winner is shown with an icon after being determined.
	Recipients	All channels and test components.	The number of contacts who received test variants and the remaining contacts who received the winning message.
	Delivered	All channels and test components.	The status is assigned as soon as Wooxy has sent a message and it has been accepted by the server of the receiving provider.
	Open Rate	All test components of the Email, Telegram, and Viber channels.	The “Unique opens” is the number of recipients who have opened the A/B Test a unique number of times.
	Opens	All test components of the WebPush channel.	The percentage of viewers who saw your Web Push notification.
	Click Rate	All channels and test components.	The “Unique clicks” is the number of recipients who have clicked the A/B Test a unique number of times.
	Unsubscribed	All test components of the Email channel.	The percentage of recipients who clicked on the unsubscribe link.
	Soft bounces	All test components of the Email channel.	This is an email that couldn’t be delivered because of a temporary reason. Common reasons an email may soft bounce include: The mailbox is full. The message is too large. The server timed out.
	Hard bounces	All test components of the Email channel.	This is an email that couldn’t be delivered for a permanent reason. Common reasons an email may hard bounce: Fake email address. The email address no longer exists. Invalid email address. Email blocked by recipient’s server.
	Marked as spam	All test components of the Email channel.	Indicates the total number of users that have marked your Email Campaign as spam in their inbox.
	Reject	All test components of the Email and Viber channels.	The message was refused due to Wooxy account restrictions: The daily limit for sending emails has been exhausted. An attempt was made to send an email to a contact that is blacklisted. An anti-spam trigger has been triggered.
	Failed	All test components of the WebPush and Viber channels.	The telegram message wasn’t delivered (if the contact unsubscribed from the bot through “stop and block bot” function). This status also typically occurs when the Viber message cannot be delivered to the recipient for various reasons: The recipient is unavailable. If the recipient has blocked your Viber account, messages from your account will not be delivered.
	Expired	All test components of the WebPush and Viber channels.	The Viber messages and WebPush notifications weren’t shown due to TTL (Time to live).

The A/B Test Details Overview

Parameter	Description
Channel	The type of channel.
Status	The current status of the A/B Test.
Sender Name	The selected Sender Name that your contacts see when they receive a Viber message from your company through the Wooxy platform.
Failure reason	Template test failed	Invalid Template. This error message appears when some changes were made in the Template before the Test launching.
	Not enough audience to test the variant	Less than 2 contacts (when 2 variants are testing) in a test environment. We recommend you have at least 5000 contacts for the A/B Test Campaign.
	Telegram Bot was deleted	Displays for two reasons: When the Telegram Bot was deleted. When the Telegram Bot is not available at the A/B Test start moment.
	Email Template was deleted	Appears when the Email Template is deleted before the A/B Test starts.
	WebPush Template was deleted	Appears when the WebPush Template is deleted before the A/B Test starts.
	Telegram Template was deleted	Appears when the Telegram Template is deleted before the A/B Test starts.
	Viber template was deleted	Appears when the Viber Template is deleted before the A/B Test starts.
	Your sender name has been paused	The Sender Name is deactivated/paused.
	No Viber sender name	Appears when the Viber Sender Name is deleted before the Viber Campaign starts.
	twig execution error	This error occurs during Template processing in the Twig syntax. To fix it, examine the error message, identify the root cause, and make the necessary corrections in your Twig Template or code.
	Contact list was deleted	Appears when the Contact List is deleted before the A/B Test starts.
	Filter variable does not exist	Appears when the variable in the selected/created filter has been deleted so then the number of contacts becomes not relevant, and the contacts do not correspond to the filter conditions.
	Undefined telegram bot	Appears when either the token is out of date or has been deleted.
	Invalid variants count	Not enough contacts have been assigned to test one of the variants. Contacts may have been deleted or they may have simply unsubscribed from the corresponding channel.
Name	The name of the A/B Test Campaign along with its ID.
Creation date	The A/B Test creation date and time. Displays according to the account’s time zone.
Template	The name of the Template along with its ID.
Template A	The names of the Templates of the A/B Test Campaign along with their IDs.
Template B
Template C
Email subject	The subject line of the email.
Subject A	The subject lines of the email that undergo the A/B Testing.
Subject B
Subject C
From name	The provided name that is displayed in your recipient’s inbox.
From name A	The From names that undergo the A/B Testing. Your clients will receive emails from the provided names.
From name B
From name C
Sender Name	The selected Sender Name that your contacts see when they receive a Viber message from your company through the Wooxy platform.
TTL (Time to live)	The defined duration of the push service attempts to deliver the notification and also how long the viber service will attempt to deliver the message.
Telegram bot	The name of the selected Telegram bot.
Contact list	The name of the selected Contact List along with its ID.
Segments	The selected segment of contacts who were sent the Campaign.
Filtered by	The one-time filter conditions used for Audience segmentation to include contacts to the Campaign.
Excluded by filter	The one-time filter conditions used for Audience segmentation to exclude contacts from the Campaign.
Launch type	The A/B Test launch type determines how the test is launched.
Launch time	The A/B Test starting time.
Launch time A	The A/B Test start time for each launch time testing variant.
Launch time B
Launch time C
Criterion & Duration	Defined criteria for choosing the winning message and selected A/B Test duration. The winner will be determined based on the data for the specified time period.
Winning message send time	The winner sending date and time. Displays when the “send winner automatically” option is selected.
Tags	The list of Tags of the A/B Test.