Control Group (Beta - release soon)
A control group is very helpful to measure the performance of a campaign. A control group is a small group of users (a percentage of users from associated segments) which is willingly not exposed to an ad or an email in order to measure the difference in terms of engagement or conversion between the population not exposed (control group) and the population exposed (users in the associated segments).
In our case, we take into consideration only conversions for now (online and offline purchases).
A control group is setup on a stream (share menu). Check the box ‘Set control group’ to activate the possibility to configure a control group.
Specify the % of users you want to set for your control group: these users will be not pushed on the stream (and so not exposed to the campaign).
10% is generally enough to have statistically significant metrics, however for very small segments it is recommended to double this %.
As soon as you start the stream, the control group will be launched also, and results will be recorded.
you can have at most 3 streams with an active control group at the same time
be careful with the modification of streams with an active control group as it impacts the results on the campaign performance dashboard. Same with segments used in a stream with an active control group, if you modify a condition it impacts all the results.
When a stream is over you cannot relaunch it, you should duplicate it
When a stream with a control group is activated, on the stream list page an icon will appear on the right of the stream’s name.
Click on this icon to access to the analytics part.
First, select the period of analysis: you can choose for example to read the results on the last 3 days or last week.
All the figures will be related to this analysis period.
Definition of each metrics:
Total population: number of users in segments which were selected on the stream.
It is equals to Audience activated + Control Group.
Audience activated: number of users that were pushed on the stream (and exposed to the campaign).
It is equals to Associated segments – control group.
Control Group: number of users in the control group, it represents the % of users selected to be in the control group and not exposed to the campaign.
It is equals to Associated segments – audience activated.
Cards
You have here some metrics related to conversions.
We compare the performance between the users exposed and not exposed in term of conversions (online and offline purchases).
In details:
Represents the impact of the campaign. As we compare users in the activated audience and users in the control group, we have to determine the impact of the campaign in term of conversions.
If the uplift is low or negative, that means users in control group have good results without the exposure to the campaign and so the campaign is not as efficient as it should be.
On contrary, if the uplift is high, that means the campaign is over performing because users exposed to the campaign convert more than users in the control group, the impact is very positive on purchases.
Formula:
Uplift = ((conversion rate ‘Audience activated’ - conversion rate ‘Control group’) ÷ conversion rate ‘Audience activated’) x 100
For each population we calculate here the conversion rate over the period selected.
Formula:
Conversion rate = number of conversions ÷ number of users
(we compare the conversion rate calculated for the population ‘audience activated’ and for the population ‘control group’)
We compare here the revenue per user for each population (audience activated and control group).
Formula:
Average Revenue Per User (ARPU) = Total revenue ÷ number of users
(we compare the ARPU calculated for the population ‘audience activated’ and for the population ‘control group’)
We compare here the average basket amount for each population.
Formula:
Average basket amount = Total revenue ÷ number of conversions
(we compare the average basket amount calculated for the population ‘audience activated’ and for the population ‘control group’)
Charts
You can see here the conversion rate evolution day after day for the two populations: the audience activated and the control group.
This chart is cumulative in order to reflect more the impact in term of users, exposure and conversions, that means for each day conversions are added.
You can visualize directly here the comparison between the two populations (audience activated and control group) for the following metrics: ARPU (Average Revenue Per User); Average Basket Amount; Conversion Rate. It allows you to analyse quickly which metrics are very impactful and relevant to compare the populations.
You have here a summary of all metrics needed to compare the populations, such as the number of users and number of conversions (needed to calculate the conversion rate for example). The turnover is based on total_final_conversion_amount, meaning it could evolve depending on products returns for example.