A/B Testing with Alteryx (Part 1)
The Business Problem
A restaurant chain, Round Roaster, is planning to introduce some new items to its Menu. But because the items are new, the restaurant is not sure of its customers' reception and the subsequent impact on sales of the New additions. Because this is a new product, they do not have data to do other types of predictive modeling. What can the restaurant do in this situation?
A/B testing to the Rescue!
What is A/B Testing?
A/B testing is a type of experiment with two variants A and B. It is a way to compare two versions of a variable to find out which performs better in a controlled environment. The two variants A and B are called the control and the variant groups respectively. Organisations use A/B testing when they are introducing a new situation without enough data for other modeling techniques. For instance, a company launching a new product, or a policy and most commonly in healthcare. In this instance, it is the introduction of a new Menu by a company, because they have not sold this project before, they are not certain what the impact of the product on their sales will be. This is a very good case for A/B testing.
A/B testing allows us to measure the impact of a new idea in the real world at minimal risk.
Broadly there are two types of A/B testing, based on how the control and treatment groups are assigned to each other, one is Randomised design and the other is Matched pair design. In Randomised designs, treatment and control units are selected randomly, and it is used in situations where there is a high volume of observations, little opportunity to control the variables and the concern for bias is low. In Matched pair design, is used when the volume of observations is fairly low, the concern for bias is high and the cost per observation is also high, as in the case of our restaurant chain.
How does A/B testing work?
To do an A/B test, we set up an experiment, by dividing our subjects into groups, usually two groups A and B as stated above, one group is administered the change you are trying to measure and the other is not. The group that is administered that change is called the treatment group while the other group is called the control group. We try to keep both groups as similar as possible so we are sure that whatever results we obtain in your experiment is due to the change we have introduced. In our case, we split the stores into two groups, one group is assigned the new menu and the other is not and then we analyze the impact on a predictive factor, in this instance Sales. Sales are referred to as a predictor variable but unlike in conventional modeling, this predictor variable is not used to predict anything but to observe the effects of the changes we are going to effect. We can say it is a predictor of the effect of the change we are introducing.
We set up our test and we run it for at least a cycle of the process we are trying to measure. A cycle is not fixed for all types of tests and it usually depends on the type of tests we are trying to implement. For example, in our store's examples, our cycle will be most likely how frequently our customers return to the store. If it's on average a month, then our test needs to be at least 1 month long. It is usually set to be a period of time to ensure that all populations in the data are captured.
A/B testing can also be defined as a test to see how a particular change to an item or situation is going to affect a particular outcome.
Some terminologies in A/B Testing:
A unit: A unit is an individual in a test, it can be a person, store, website, etc. A unit is an item on which the change or the test is going to be administered. In our case, a unit is a store.
Treatment group: The group that gets the treatment or the change is called the treatment group.
Control group: The control group does not get the treatment and is used to compare the treatment. Control units are the units that will be matched to the treatment units using the control variables.
Control variables: Control variables are features that we are going to keep constant in our experiment, they remain unchanged in both the treatment and control groups. So in the case of our store, we will use the control variables to match control units to treatment units that are as close as possible.
Tools for A/B Testing
There are different tools to carry out A/B testing with different degrees of complexity. Some examples are:
Microsoft
Excel
Python
R
Alteryx
Knime and so many other tools.
Alteryx
I will be using Alteryx for this experiment. Alteryx is a low-code/no-code tool for data science and analytics. You drag and drop using the graphic user interface and you can connect to data from a variety of sources. It has a wide range of functions and it is easy to use. I find it to be a great and easy-to-use tool, especially for statistical analysis like A/B testing and time series. It does have some downsides, some of which are that it is expensive and presently only available on windows. I used the designer version for this project and it costs $5195 per annum. If you are a student or an educator, they have a free 1-year license for students and educators. There is also a free 1 month trial period, which you can explore here; Alteryx has many tools for A/B testing which we will be using for this project, they are:
AB treatment Tool: This tool helps to select the treatment groups for the test
AB controls tool: The controls tool matches the treatment group to the control group.
AB trend tool: This tool looks for patterns in the data to match the two groups
AB analysis tool: This tool analyzes the results and comparison for the tests.
An example of a workflow in alteryx Tridant.com[https://www.tridant.com/blog/introduction-to-alteryx/]
These tools make it very easy to set up and analyse your test. You don't have to set up your calculations manually and you can customise the tools to your needs.
In the second part of this series on Alteryx, I walkthrough solving the business problem here using AB Testing on Alteryx.