A|B Testing

A|B Testing

A|B Testing and its Impact on Data-driven Decision

As the competitive landscape continues to evolve and leveraging actionable insights from your data becomes increasingly necessary to build successful business models, your company will face big decisions on market opportunities, strategy and more. Overcoming these challenges will require more than simple instinct and past experience. It also will require an ability to create data-driven strategies and to execute them efficiently.

What is A|B Testing

A|B Testing is a common approach toward optimization that can lead to controlled improvement throughout the lifecycle of a process or product. In its most basic form it consists of splitting your population ( clients, employees, visitors etc..) into two distinct groups, one would be called the control group and the other the treatment group. It is important to ensure that both groups must hold statistical significance in order for your results to make sense.

Having split both groups you will administer the treatment group with the improvement you wish to test. This is usually determined by a hypothesis and is backed up by certain analytical and numerical studies of the problem. The control group, on the other hand, will receive no treatment, this will function as your baseline. In both cases, you need to define the KPI with which you will be measuring the success of your initiative. This can be conversion, engagement, client satisfaction spend etc..

Dependent & Independent Variables

You KPI’s will be either continuous or discrete and will be classified under two categories, dependent and independent

  • Independent variables: are exactly what it sounds like. These variables’ values do not depend on the value of another variable, they can be considered as fixed. An example of this is Age, a client’s age will not change based how often they visit a site, what they eat, or what school do they go to
  • Dependent Variable: again like the independent variable dependent variables exactly what it sounds like. These variables are dependent on another variable or factor. An example of this is client conversion, as this may be affected by a number of factors such as Age, Sex, time spent shopping etc.

An easy way to test whether a variable is independent or dependent is to fill the following sentence:

(Independent variable) causes a change in (Dependent Variable) and it isn’t possible that (Dependent Variable) could cause a change in (Independent Variable).

For example:

(Time Spent Studying) causes a change in (Test Score) and it isn’t possible that (Test Score) could cause a change in (Time Spent Studying).

This would sound kind-of off the other way around.

Statistical Significance

After running your experiment for a given period of time, you need to determine whether or not you have reached statistical significance with your results. This basically means that if you were to replicate this experiment, would you with confidence be able to expect a similar result. This is most easily achieved by using a statistical calculator.


Having analyzed statistical significance and the results of your experiment you will reach one of three conclusions:

  • Implement: The experiment was a success and should be further replicated through the control group
  • Abandon: The experiment proved the hypothesis to be wrong and should be discontinued
  • Inconclusive. You weren’t able to reach statistical significance to reach conclusive answers and further testing is required.

Regardless of the outcome, it is important to document the results of your test, this will help future testers to have a good idea of the Do’s and Dont’s and can help save time in money in implementing future Ideas.


Need Help Getting Started?

We can help your team start A|B testing and driving improvement through a data-backed process. If you want to know how you can discover new value and gain an edge in the rapidly evolving market we should talk.

Looking for a First-Class Business Plan Consultant?