Two-Active-Treatment (Bivariate A/B) Controlled Experiment

From GM-RKB
(Redirected from bucket testing)
Jump to navigation Jump to search

A Two-Active-Treatment (Bivariate A/B) Controlled Experiment is a active treatment-controlled experiment with two active treatments.



References

2020

  • (Optimizely, 2020) ⇒ https://www.optimizely.com/optimization-glossary/multivariate-test-vs-ab-test/
    • QUOTE: ... What is the difference between A/B testing and multivariate testing? Let's take a look at the methodology, common uses, advantages, and limitations of these testing methods.
      • A/B testing, which you may also have heard referred to as split testing, is a method of website optimization in which the conversion rates of two versions of a page — version A and version B — are compared to one another using live traffic. Site visitors are bucketed into one version or the other. ...

        ...

      • Multivariate testing uses the same core mechanism as A/B testing, but compares a higher number of variables, and reveals more information about how these variables interact with one another. As in an A/B test, traffic to a page is split between different versions of the design. The purpose of a multivariate test, then, is to measure the effectiveness each design combination has on the ultimate goal. ...

        ...

2018

2017

2016a

  • (Wikipedia, 2016) ⇒ https://en.wikipedia.org/wiki/A/B_testing Retrieved:2016-9-14.
    • In marketing and business intelligence, A/B testing is a term for a randomized experiment with two variants, A and B, which are the control and variation in the controlled experiment. A/B testing is a form of statistical hypothesis testing with two variants leading to the technical term, two-sample hypothesis testing, used in the field of statistics. Other terms used for this method include bucket tests and split-run testing. These terms can have a wider applicability to more than two variants, but the term A/B testing is also frequently used in the context of testing more than two variants. In online settings, such as web design (especially user experience design), the goal of A/B testing is to identify changes to web pages that increase or maximize an outcome of interest (e.g., click-through rate for a banner advertisement). Formally the current web page is associated with the null hypothesis. A/B testing is a way to compare two versions of a single variable typically by testing a subject's response to variable A against variable B, and determining which of the two variables is more effective. As the name implies, two versions (A and B) are compared, which are identical except for one variation that might affect a user's behavior. Version A might be the currently used version (control), while version B is modified in some respect (treatment). For instance, on an e-commerce website the purchase funnel is typically a good candidate for A/B testing, as even marginal improvements in drop-off rates can represent a significant gain in sales. Significant improvements can sometimes be seen through testing elements like copy text, layouts, images and colors, but not always. The vastly larger group of statistics broadly referred to as multivariate testing or multinomial testing is similar to A/B testing, but may test more than two different versions at the same time and/or has more controls, etc. Simple A/B tests are not valid for observational, quasi-experimental or other non-experimental situations, as is common with survey data, offline data, and other, more complex phenomena. A/B testing has been marketed by some as a change in philosophy and business strategy in certain niches, though the approach is identical to a between-subjects design, which is commonly used in a variety of research traditions. A/B testing as a philosophy of web development brings the field into line with a broader movement toward evidence-based practice. The benefits of A/B testing are considered to be that it can be performed continuously on almost anything, especially since most marketing automation software now, typically, comes with the ability to run A/B tests on an on-going basis. This allows for updating websites and other tools, using current resources, to keep up with changing trends.


2016b

Assumed Distribution Example Case Standard Test Python Implementation
Gaussian Average Revenue Per Paying User Welch's t test scipy.stats.ttest_ind
Binomial Click Through Rate Fisher's exact test scipy.stats.fisher_exact
Poisson Average Transactions Per Paying User E-test None
Multinomial Number of each product Purchased Chi-squared test scipy.stats.chisquare
Unknown -- Mann–Whitney U test scipy.stats.mannwhitneyu

2006