A/B Testing Task

Jump to: navigation, search

An A/B Testing Task is a controlled experiment testing task that creates an A/B test.



  • http://hookedondata.org/Guidelines-for-AB-Testing/
    • QUOTE: ... The concept of A/B Testing seems pretty simple. A classic example is you change the color of a button and measuring if the click-rate changes. Assuming your assignment of visitors and data collection is working, all you need to do is run a proportion test, right? And if you already have the proportion test calculated, why is a data scientist even needed? Maybe you need one if you want to do some fancy techniques like multi-armed bandits, but how can classic, frequentist A/B Testing be a challenge? Unfortunately, “generating numbers is easy; generating numbers you should trust is hard!” There’s many ways A/B Testing can go wrong, but most of them won’t be obvious.

      This post outlines some recommended best practices for A/B Testing. I’ve found that a lot of analysts and data scientists struggle with A/B testing, especially those not classically trained in statistics or who are trying to start their company’s A/B testing system. While A/B testing correctly isn’t easy, these 12 guidelines will help you guard against some common mistakes and set you up for success. ...



  • (Wikipedia, 2011) ⇒ http://en.wikipedia.org/wiki/A/B_testing
    • QUOTE: A/B testing is a methodology of using randomized experiments with two variants, A and B, which are the Control and Treatment in the controlled experiment. Such experiments are commonly used in web development and marketing, as well as in more traditional forms of advertising. Other names include randomized controlled experiments, online controlled experiments, and split testing. In online settings, such as web design (especially user experience design), the goal is to identify changes to web pages that increase or maximize an outcome of interest (e.g., click-through rate for a banner advertisement). As the name implies, two versions (A and B) are compared, which are identical except for one variation that might impact a user's behavior. Version A might be the currently used version (Control), while Version B is modified in some respect (Treatment). For instance, on an e-commerce website the purchase funnel is typically a good candidate for A/B testing, as even marginal improvements in drop-off rates can represent a significant gain in sales. Significant improvements can be seen through testing elements like copy text, layouts, images and colors.[1] Multivariate testing or bucket testing is similar to A/B testing, but tests more than two different versions at the same time.

      While the approach is identical to a between-subjects design, which is commonly used in a variety of research traditions, A/B testing is seen as a significant change in philosophy and business strategy in Silicon Valley.[2][3][4] A/B testing as a philosophy of web development brings the field into line with a broader movement toward evidence-based practice