Coming from a process (optimal) control background, I have begun studying the field of deep reinforcement learning.

Sutton & Barto (2015) state that

particularly important (to the writing of the text) have been the contributions establishing and developing the relationships to the theory of optimal control and dynamic programming

With emphasis on the elements of reinforcement learning - that is, policy, agent, environment, etc., what are the key differences between (deep) RL and optimal control theory?

In optimal control we have, controllers, sensors, actuators, plants, etc, as elements. Are these different names for similar elements in deep RL? For example, would an optimal control plant be called an environment in deep RL?

  • $\begingroup$ A classical optimal control policy was created manually. In the system identification process the forward dynamics of a plant is realized in an equation and then the forward model is used to bring the plant into the goal state. In contrast, reinforcement learning optimizes itself automatically. Usually, no sourcecode programming is needed, instead a q-table or a neural network adapts it's parameters. Reinforcement learning can be combined with optimal control. This is a recent idea described in the Arxiv-literature since 2017. $\endgroup$ – Manuel Rodriguez Mar 22 '19 at 11:04
  • $\begingroup$ Thank you Manuel for your quick response. I have noticed many arXiv articles mention reinforcement learning and control for HVAC systems. In this case, would the RL agent simply replace the temperature controller system (i.e. PID controller and actuators)? $\endgroup$ – Bionic Buffulo Mar 22 '19 at 11:11
  • $\begingroup$ Have a look at this Quora question: qr.ae/TWpKtm. It contains several possibly useful answers. $\endgroup$ – nbro Apr 21 '19 at 13:13

Classical reinforcement learning which was explained in the Arxiv-papers from 2000 to 2015 is equal to model-free learning. A problem was given, for example to solve a simple game, and the agent's policy gets learned in the q-matrix. Alternatively, a neural network was used to described the direct policy of the agent in the system. The so called state-action space of a q-table defines, what the agent has to do in each situation. The agent gets as sensory input the current situation, selects by it's learned policy the best action, and then the action is executed.

The inner working of the classical reinforcement learning paradigm stays in contrast to what the optimal control literature describes. Optimal control and model predictive control is a model-based strategy which is testing out different actions against a forward model and selects the best action as a result of the simulation experiment. The classical optimal control literature published in the 1990s is working with handcrafted forward models which are describing a plant. For example, a housing heater can be described with differential equations in a Matlab script. This forward model is used by the receding horizon controller which brings the system into a goal state. Model predictive control doesn't contain a direct policy, but it's using a predictive controller.

In the classical literature until the year 2015 the difference was made, that optimal control is a model based control technique which is working with a handcrafted forward model while reinforcement learning is model-free and it is based on a learned policy. The situation changed since the year 2017 in which many new papers were published on arxiv which are trying to combine both principles. So called model-based reinforcement learning is using a forward model, stored in a neural network to bring a system into a goal state.

| improve this answer | |
  • $\begingroup$ The difference you mentioned here makes sense, especially in the context of the similarities stated in the text. Sutton & Barto (2015) "use the terms agent, environment, and action instead of the engineers’ terms controller, controlled system (or plant), and control signal because they are meaningful to a wider audience." $\endgroup$ – Bionic Buffulo Mar 26 '19 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.