The contents in policy optimization are introduced in two main categories: value-based optimization and policy-based optimization. We further introduce deep reinforcement learning for both policy and value function approximation in policy optimization. Three main approaches for solving the Bellman equations are then introduced: dynamic programming, Monte Carlo method, and temporal difference learning. The optimal value functions and optimal policy can be derived through solving the Bellman equations. The combination of the Markov reward process and value function estimation produces the core results used in most reinforcement learning methods: the Bellman equations. These notions are the cornerstones in formulating reinforcement learning tasks. Next, we introduce the Markov process, together with the Markov reward process and the Markov decision process. Then, we describe a classical reinforcement learning problem, the bandit problem, to provide the readers with a basic understanding of the underlying mechanism of traditional reinforcement learning. We first start with the basic definitions and concepts of reinforcement learning, including the agent, environment, action, and state, as well as the reward function. In this chapter, we introduce the fundamentals of classical reinforcement learning and provide a general overview of deep reinforcement learning.

Solid converter pdf v2 2.158