1: \begin{proof}

2: % We first prove that the left-hand side of the theorem derives the right-hand side. From $b(\vect{\theta}+\lrate (\vect{o}'-\beta b(\theta)\vect{b}'))<b(\vect{\theta})$ with a sufficiently small $\lrate > 0$, we have

3: % \begin{align*}

4: %     \lim_{\lrate\to 0^{+}} \frac{ b(\vect{\theta}+\lrate (\vect{o}'-\beta b(\theta)\vect{b}')) - b(\vect{\theta})}{\lrate}<0.

5: % \end{align*}

6: % Similarly, if we assume $b(\vect{\theta}+\lrate (\vect{o}'-\beta b(\theta)\vect{b}'))>b(\vect{\theta})$ with a sufficiently small $\lrate < 0$, we have

7: % \begin{align*}

8: %     \lim_{\lrate\to 0^{-}} \frac{ b(\vect{\theta}+\lrate (\vect{o}'-\beta b(\theta)\vect{b}')) - b(\vect{\theta})}{\lrate}<0.

9: % \end{align*}

10: 

11: % $\lrate$ should be positive (resp. negative) if gradient ascent (resp. descent). We mention both of $\lrate\to 0^{+}$ and $\lrate\to 0^{-}$ in this proof so our theorem is the case for both the gradient ascent and descent, which makes our proof rigorous. These indicate that the directional derivative %\footnote{See Appendix for the concept of directional derivative.} 

12: % of $b(\vect{\theta})$ along $\vect{o}' - \beta b(\theta)\vect{b}'$ is negative.

13: 

14: % Following the definition of directional derivative, we have

15: % \begin{align*}

16: % 	    \nabla_{\vect{o}' - \beta b(\theta)\vect{b}'} b(\vect{\theta}) \equiv \nabla b(\vect{\theta}) \cdot \big(\vect{o}' - \beta b(\theta)\vect{b}'\big) \equiv \vect{b}' \cdot \big(\vect{o}' - \beta b(\theta)\vect{b}'\big).

17: % \end{align*}

18: % Thus, we obtain

19: % \begin{align*}

20: %     \vect{b}' \cdot \big(\vect{o}'-\beta b(\theta)\vect{b}'\big) < 0.

21: % \end{align*}

22: % Then,

23: % \begin{align*}

24: %     \beta > \frac{\vect{b}' \cdot \vect{o}'}{b(\theta)\vect{b}' \cdot \vect{b}'}.

25: % \end{align*}

26: % The opposite direction from the right-hand side to the left-hand side is straightforward by reversing the deduction process.

27: % \end{proof}

28: