Control Group (Beta - release soon)

What is a control group?

A control group is very helpful to measure the performance of a campaign. A control group is a small group of users (a percentage of users from associated segments) which is willingly not exposed to an ad or an email in order to measure the difference in terms of engagement or conversion between the population not exposed (control group) and the population exposed (users in the associated segments).

In our case, we take into consideration only conversions for now (online and offline purchases).

How it works?

A control group is setup on a stream (share menu). Check the box ‘Set control group’ to activate the possibility to configure a control group.

Specify the % of users you want to set for your control group: these users will be not pushed on the stream (and so not exposed to the campaign).

10% is generally enough to have statistically significant metrics, however for very small segments it is recommended to double this %.

As soon as you start the stream, the control group will be launched also, and results will be recorded.

⚠️ Limitations

• be careful with the modification of streams with an active control group as it impacts the results on the campaign performance dashboard. Same with segments used in a stream with an active control group, if you modify a condition it impacts all the results.

• When a stream is over you cannot relaunch it, you should duplicate it

How can I read the results?

When a stream with a control group is activated, on the stream list page an icon will appear on the right of the stream’s name.

Click on this icon to access to the analytics part.

First, select the period of analysis: you can choose for example to read the results on the last 3 days or last week.

All the figures will be related to this analysis period.

Definition of each metrics:

Total population: number of users in segments which were selected on the stream. It is equals to Audience activated + Control Group.

Audience activated: number of users that were pushed on the stream (and exposed to the campaign). It is equals to Associated segments – control group.

Control Group: number of users in the control group, it represents the % of users selected to be in the control group and not exposed to the campaign. It is equals to Associated segments – audience activated.

Cards

You have here some metrics related to conversions.

We compare the performance between the users exposed and not exposed in terms of conversions (online and offline purchases).

In details:

Incremental revenue

Incremental revenue represents the campaign impact on your revenue.

All starts with a supposition: if we consider the full population in the control group what are the performances, and what is the incremental revenue generated by the campaign?

This incremental revenue could be positive or negative, and it represents the campaign performance and impact.

In details, we consider the full population of users (= Audience activated + users in control group) and we apply control group results for Conversion Rate and Average Basket Amount to all these users:

Then, we can easily compare the revenue generated by the Audience activated and the simulated revenue generated by the total population as control group and see if the incremental revenue is negative or positive.

Formula:

Incremental revenue = (Revenue generated by the Audience activated + Revenue generated by the Control group) - (simulated Revenue generated by the total population as control group)

Conversion rate

For each population, we calculate here the conversion rate over the period selected.

Formula:

Conversion rate = number of conversions ÷ number of users

(we compare the conversion rate calculated for the population ‘audience activated’ and for the population ‘control group’)

Uplift (% difference for the conversion rate)

Represents the impact of the campaign. As we compare users in the activated audience and users in the control group, we have to determine the impact of the campaign in terms of conversions.

If the uplift is low or negative, that means users in control group have good results without the exposure to the campaign and so the campaign is not as efficient as it should be.

On contrary, if the uplift is high, that means the campaign is over performing because users exposed to the campaign convert more than users in the control group, the impact is very positive on purchases.

Formula:

Uplift = ((conversion rate ‘Audience activated’ - conversion rate ‘Control group’) ÷ conversion rate ‘Audience activated’) x 100

Average Revenue Per User (ARPU)

We compare here the revenue per user for each population (audience activated and control group).

Formula:

Average Revenue Per User (ARPU) = Total revenue ÷ number of users

(we compare the ARPU calculated for the population ‘audience activated’ and for the population ‘control group’)

Average basket amount

We compare here the average basket amount for each population.

Formula:

Average basket amount = Total revenue ÷ number of conversions

(we compare the average basket amount calculated for the population ‘audience activated’ and for the population ‘control group’)

Charts

Conversion rate evolution

You can see here the conversion rate evolution day after day for the two populations: the audience activated and the control group.

This chart is cumulative in order to reflect more the impact in terms of users, exposure and conversions, that means for each day conversions are added.

Radar

You can visualize directly here the comparison between the two populations (audience activated and control group) for the following metrics: ARPU (Average Revenue Per User); Average Basket Amount; Conversion Rate. It allows you to analyze quickly which metrics are very impactful and relevant to compare the populations.

Table

You have here a summary of all metrics needed to compare the populations, such as the number of users and number of conversions (needed to calculate the conversion rate for example). The turnover is based on total_final_conversion_amount, meaning it could evolve depending on products returns, for example.

Can I export the control group?

Yes, you can select an export (email, S3 or FTP for example), and select the universe 'campaign':

You can select 1 segment or choose 'all users' and you can export the following variables:

• Control Group: does the user belong to the control group? True / False

• Exposure count: number of exposure to campaign for a user = number of times a user entered a segment and was pushed through the connector

• First exposure: date of first exposure

• Last exposure: date of last exposure

• Stream ID: ID of the stream for which the control group was activated