1: \begin{abstract}

2: When has an agent converged?

3: %

4: %

5: Standard models of the reinforcement learning problem give rise to a straightforward definition of convergence: An agent converges when its behavior or performance in each environment state stops changing.

6: %

7: %

8: However, as we shift the focus of our learning problem from the environment's state to the agent's state, the concept of an agent's convergence becomes significantly less clear.

9: %

10: %

11: % This paper.

12: In this paper, we propose two complementary accounts of agent convergence in a framing of the reinforcement learning problem that centers around bounded agents.

13: %

14: %

15: % First: performance.

16: The first view says that a bounded agent has converged when the minimal number of states needed to describe the agent's future behavior cannot decrease.

17: %

18: %

19: % Second: behavior.

20: The second view says that a bounded agent has converged just when the agent's performance only changes if the agent's internal state changes.

21: %

22: %

23: % Relationship of two proposals.

24: We establish basic properties of these two definitions, show that they accommodate typical views of convergence in standard settings, and prove several facts about their nature and relationship.

25: %

26: %

27: % Impact.

28: We take these perspectives, definitions, and analysis to bring clarity to a central idea of the field.

29: \end{abstract}

30: