7
$\begingroup$

Coming from a process (optimal) control background, I have begun studying the field of deep reinforcement learning.

Sutton & Barto (2015) state that

particularly important (to the writing of the text) have been the contributions establishing and developing the relationships to the theory of optimal control and dynamic programming

With an emphasis on the elements of reinforcement learning - that is, policy, agent, environment, etc., what are the key differences between (deep) RL and optimal control theory?

In optimal control we have, controllers, sensors, actuators, plants, etc, as elements. Are these different names for similar elements in deep RL? For example, would an optimal control plant be called an environment in deep RL?

$\endgroup$
  • $\begingroup$ Most optimal control books focus on deterministic systems. RL is generally applied to systems which are probabilistic. $\endgroup$ – FourierFlux Sep 4 at 22:22
1
$\begingroup$

The same book Reinforcement learning: an introduction (2nd edition, 2018) by Sutton and Barto has a section, 1.7 Early History of Reinforcement Learning, that describes what optimal control is and how it is related to reinforcement learning. I will quote the most relevant part to answer your question, but you should read all that section to have a full understanding of the relationship between optimal control and reinforcement learning.

The term "optimal control" came into use in the late 1950s to describe the problem of designing a controller to minimize or maximize a measure of a dynamical system's behavior over time. One of the approaches to this problem was developed in the mid-1950s by Richard Bellman and others through extending a nineteenth-century theory of Hamilton and Jacobi. This approach uses the concepts of a dynamical system's state and of a value function, or "optimal return function", to define a functional equation, now often called the Bellman equation. The class of methods for solving optimal control problems by solving this equation came to be known as dynamic programming (Bellman, 1957a). Bellman (1957b) also introduced the discrete stochastic version of the optimal control problem known as Markov decision processes (MDPs). Ronald Howard (1960) devised the policy iteration method for MDPs. All of these are essential elements underlying the theory and algorithms of modern reinforcement learning.

To answer your specific questions.

In optimal control we have, controllers, sensors, actuators, plants, etc, as elements. Are these different names for similar elements in deep RL? For example, would an optimal control plant be called an environment in deep RL?

Yes. In reinforcement learning (see the first footnote of the cited book on page 48), the term control is often used as a synonym for action. Similarly, the term controller is used as a synonym for agent (and I would say it is also a synonym for policy, given that the policy usually defines and controls the agent, although the concept of the agent is more abstract and we could associate more than one policy with the same agent). The term environment is also used as a synonym for controlled system (or plant).

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I think the controller is synonym for agent, at least Sutton & Barto agrees with this "We use the terms agent, environment, and action instead of the engineers’ terms controller, controlled system (or plant), and control signal because they are meaningful to a wider audience." (from incompleteideas.net/book/RLbook2020.pdf footnote p48) $\endgroup$ – Neil Slater Sep 4 at 22:24
  • $\begingroup$ @NeilSlater Thanks for noticing this. I was actually looking for that footnote (that I knew existed because I had answered another question based on it), but, for some reason, this time I didn't find it. Anyway, many times, a policy defines and controls the agent, so I would say that a policy is a controller, although an agent is a more abstract concept and we could indeed associate more than one policy with the same agent. $\endgroup$ – nbro Sep 4 at 22:37
0
$\begingroup$

As a supplement to nbro's nice answer, I think a major difference between RL and optimal control lies in the motivation behind the problem you're solving. As has been pointed out by comments and answers here (as well as the OP), the line between RL and optimal control can be quite blurry.

Consider the LQG algorithm (linear quadratic gaussian) which is generally considered to be an optimal control method. Here a controller is computed given a stochastic model of the environment and a cost function.

Now consider AlphaZero, which is obviously thought of as an RL algorithm. AlphaZero learns a value function (and thus a policy/controller) in a perfect information setting with a known deterministic model.

So it's not the stochasticity that separates RL from optimal control as some people believe. It's also not the presence of a known model. I argue that the difference between RL and optimal control comes from the generality of the algorithms.

For instance, generally when applying LQG and other optimal control algorithms, you have a specific environment in mind and the big challenge is modeling the environment and the reward function to achieve the desired behavior. In RL on the other hand, the environment is generally thought of as a sort of black box. While in the case of AlphaZero the model of the environment is known, the reward function itself was not designed specifically for the game of chess (for instance, it's +1 for win and -1 for loss regardless of chess, go, etc). Furthermore the neat thing with AlphaZero is that we can use it to train agents in virtually any perfect information game without changing the algorithm at all. Another difference with RL here is that the agent iteratively improves itself, while optimal control algorithms learn controllers offline and then stay fixed.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Rather than saying "the environment is generally thought of as a sort of black box", I think you want to say that you don't have the model of environment's dynamics (i.e. the transition function/probability distribution). So, it may be better to be more specific. You may also want to mention "planning" and "dynamic programming" algorithms and how they relate to your description of "optimal control". $\endgroup$ – nbro Oct 5 at 21:34
  • $\begingroup$ That's actually not what I'm saying though -- in AlphaZero you do have a dynamics model. My point was that in optimal control your task depends on how you model things, while in RL you have no control over it. $\endgroup$ – harwiltz Oct 5 at 22:11
  • $\begingroup$ I am not sure if this is what you also mean, but in RL you typically have control over the reward function, so, in this sense, you can control the task. $\endgroup$ – nbro Oct 5 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.