1: \begin{abstract}

2: I examine a conceptual model of a recommendation system (RS) with user inflow and churn dynamics. 

3: When inflow and churn balance out, the user distribution reaches a steady state. Changing the recommendation algorithm alters the steady state and creates a transition period. During this period, the RS behaves differently from its new steady state. 

4: In particular, A/B experiment metrics obtained in transition periods are biased indicators of the RS’s long term performance. 

5: Scholars and practitioners, however, often conduct A/B tests shortly after introducing new algorithms to validate their effectiveness. 

6: This A/B experiment paradigm, widely regarded as the gold standard for assessing RS improvements, may consequently yield false conclusions. 

7: I also briefly discuss the data bias caused by the user retention dynamics.

8: 

9: \medskip

10: 

11: \noindent\textsc{Keywords}: recommendation system, A/B experimentation, bias, user retention and churn, system convergence

12: 

13: \end{abstract}

14: