1: \begin{abstract}

2: In this paper, we draw attention to a problem that is often overlooked or ignored

3: by companies practicing hypothesis testing (A/B testing) in online environments.

4: We show that conducting experiments on limited inventory that is shared between variants in the experiment

5: can lead to high false positive rates since the core assumption of independence between the groups is violated. We provide a detailed analysis of the problem in a simplified setting whose parameters are informed by realistic scenarios.

6: The setting we consider is a $2$-dimensional random walk in a semi-infinite strip. It is rich enough to take a finite inventory into account, but is at the same time simple enough to allow for a closed form of the false-positive probability.

7: We prove that high false-positive rates can occur, and develop tools that are suitable to help design adequate tests in follow-up work.

8: Our results also show that high false-negative rates may occur. The proofs rely on a functional limit theorem for the $2$-dimensional random walk in a semi-infinite strip.

9: \end{abstract}

10: