Online controlled experimentation (OCE), also called A/B testing, is an often used tool in industry to determine if deploying changes into production is the right decision to make. Running experiments has shown an immense impact to the revenue of companies in industry, however this type of experimentation comes with a lot of pitfalls, some which can invalidate the entire experiment. As more organizations start relying on online experimentation, relying only on experts in OCE’s becomes infeasible and creating guidance on the platform where the experiments take becomes a mandatory item.
This thesis describes the impact these pitfalls have on the work of experimenters at ING, a global bank, by performing interviews with practitioners and performing a survey with 52 participants. Next, building on existing solutions, a set of solutions is proposed to solve these pitfalls. To determine if these solutions solve the problem and will help the experimenter, in the same survey these solutions are validated. Finally, this work tries to determine how these solutions can be evaluated in a real world scenario.
This thesis shows that experimenters are well informed about the existence of pitfalls and believe that almost all should be resolved. There are many promising solution to these pitfalls which experimenters rate as helpfull. However evaluating these solutions at this point in the current context is not possible.