1: \begin{abstract}

2: We study optimal variance reduction solutions for 

3: count and ratio metrics in online controlled experiments.

4: Our methods leverage flexible machine learning tools to incorporate covariates that are independent from the treatment but have predictive power for the outcomes, and employ 

5: the cross-fitting technique to remove the bias in complex machine learning models. 

6: We establish CLT-type asymptotic inference based on our estimators under mild convergence conditions. 

7: Our procedures 

8: are optimal (efficient) for the corresponding targets

9: as long as the machine learning estimators are consistent, 

10: without any requirement for their convergence rates. 

11: In complement to the general optimal procedure, 

12: we also derive a linear adjustment method for ratio metrics 

13: as a special case  that 

14: is computationally efficient and can flexibly incorporate any pre-treatment covariates. 

15: We evaluate the proposed variance reduction procedures with

16: comprehensive simulation studies and provide  practical suggestions regarding 

17: commonly adopted assumptions in computing ratio metrics. 

18: When tested on real online experiment data from LinkedIn, 

19: the proposed optimal procedure for ratio metrics can reduce 

20: up to 80\% of variance compared to the standard difference-in-mean estimator and also further reduce up to 30\% of variance compared to the CUPED approach by going beyond linearity and incorporating a large number of extra covariates.

21: \end{abstract}

22: