Informed Bayesian Inference for the A/B Test

Informed Bayesian Inference for the A/B Test

: model comparison, Bayes factor, prior elicitation, Bayesian estimation.

Booming in business and a staple analysis in medical trials, the A/B test assesses
the effect of an intervention or treatment by comparing its success rate with that of a
control condition. Across many practical applications, it is desirable that (1) evidence
can be obtained in favor of the null hypothesis that the treatment is ineffective; (2)
evidence can be monitored as the data accumulate; (3) expert prior knowledge can be
taken into account. Most existing approaches do not fulfill these desiderata. Here we
describe a Bayesian A/B procedure based on Kass and Vaidyanathan (1992) that allows
one to monitor the evidence for the hypotheses that the treatment has either a positive
effect, a negative effect, or, crucially, no effect. Furthermore, this approach enables one to
incorporate expert knowledge about the relative prior plausibility of the rival hypotheses
and about the expected size of the effect, given that it is non-zero. To facilitate the wider
adoption of this Bayesian procedure we developed the abtest package in R. We illustrate
the package options and the associated statistical results with a fictitious business example
and a real data medical example

Quentin F. Gronau

https://www.jstatsoft.org/article/view/v100i17

University of Amsterdam

November 2021

Fajar bagus W

PDF

English

Text