1: \begin{abstract}

2: %We focus on such counterfactual inferences in the context of

3: %sequential decision making in a Markov decision process, and consider how to perform

4: We study the problem of off-policy

5: policy optimization in Markov decision processes, and develop a novel off-policy policy gradient method. %Policy gradient methods have had great

6: %recent success when used in online reinforcement learning, and can be often a nice way to encode inductive

7: %bias, as well as to be able to tackle continuous action domains.

8: Prior off-policy policy gradient approaches have generally ignored the mismatch between the distribution of

9: states visited under the behavior policy used to collect data, and what would

10: be the distribution of states under the learned policy. %, in part due to the

11: %potential exponential blow up in the variance that can occur.

12: Here we build on recent

13: progress for estimating the ratio of the state distributions under behavior and evaluation policies for 

14: policy evaluation, and present an off-policy policy gradient optimization technique

15: that can account for this mismatch in distributions. We present an illustrative example

16: of why this is important and a theoretical convergence guarantee for our approach. Empirically, we compare our method in simulations to several strong baselines which do not correct for this mismatch, significantly improving in the quality of the policy discovered.

17: %We hope this is a step towards practical algorithms that can efficiently leverage

18: %prior data in order to inform better future decision policies.

19: \end{abstract}

20: