23
$\begingroup$

When designing solutions to problems such as the Lunar Lander on OpenAIGym, Reinforcement Learning is a tempting means of giving the agent adequate action control so as to successfully land.

But what are the instances in which control system algorithms, such as PID controllers, would do just an adequate job as, if not better than, Reinforcement Learning?

Questions such as this one do a great job at addressing the theory of this question, but do little to address the practical component.

As an Artificial Intelligence engineer, what elements of a problem domain should suggest to me that a PID controller is insufficient to solve a problem, and a Reinforcement Learning algorithm should instead be used (or vice versa)?

Improve this question
$\endgroup$
5
  • $\begingroup$ The basic idea I have about PID says its not easy to design. It has lots of integrals and differentials involved. So this is basically the same idea as when you replace statistics with ML approaches. Control systems is definitely flawless but it is too much work. $\endgroup$
    – user9947
    May 22, 2019 at 17:26
  • 2
    $\begingroup$ actually it's not too much work, its pretty standard in industry, using modern system design tools like MATLAB you can tune PID or any other controller relatively easy to satisfy your needs. Reinforcement learning is not applied in practice since it needs abundance of data and there are no theoretical garanties like there is for classic control theory. By the way, contoller design does not involve working directly with integrals/differentials, for linear systems all the work is done in Laplace domain which involves simple algebraic manipulations $\endgroup$
    – Brale
    May 22, 2019 at 18:27
  • $\begingroup$ @Brale_ but it still involves a lot of theoretical knowledge..Laplace domain only simplifies the differential but you need to know how to design things (poles and zeros) such that systems don't become unstable. It's pretty tough to visualize to me how those things actually work. $\endgroup$
    – user9947
    May 22, 2019 at 19:06
  • 4
    $\begingroup$ As a good rule of thumb that helped me in past projects, if you cant explain explain the optimal policy (PID, RL, or otherwise) in a few sentences, PIDs will be really really hard. What's the optimal policy for Pacman? $\endgroup$
    – Jaden Travnik
    May 22, 2019 at 22:39
  • $\begingroup$ Wouldn't it be great to lay a comparison between MPC (Model Predictive Control) and RL too ? $\endgroup$
    – Pe Dro
    Aug 21, 2022 at 3:06

1 Answer 1

Reset to default
10
$\begingroup$

I think the comments are basically on the right track.

PID controllers are useful for finding optimal policies in continuous dynamical systems, and often these domains are also used as benchmarks for RL, precisely because there is an easily derived optimal policy. However, in practice, you'd obviously prefer a PID controller for any domain in which you can easily design one: the controller's behaviors are well understood, while RL solutions are often difficult to interpret.

Where RL shines is in tasks where we know what good behaviour looks like (i.e., we know the reward function), and we know what sensor inputs look like (i.e. we can completely and accurately describe a given state numerically), but we have little or no idea what we actually want the agent to do to achieve those rewards.

Here's a good example:

  • If I wanted to make an agent to maneuver a plane from in front of an enemy plane with known movement patterns to behind it, using the least amount of fuel, I'd much prefer to use a PID controller.

  • If I wanted to make an agent to control a plane and shoot down an enemy plane with enough fuel left to land, but without a formal description of how the enemy plane might attack (perhaps a human expert will pilot it in simulations against our agent), I'd much prefer RL.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Nice example. Can u also include MPC in this example ? $\endgroup$
    – Pe Dro
    Aug 21, 2022 at 3:07
  • $\begingroup$ Personal example: in one of the labs I used to work in, years of halfhearted work with PID control had failed to achieve anything impressive to electrically stimulate human arms to go from a start position to a target position. Reinforcement learning quickly discovered much better control, since we just weren't good at applying control theory well enough. Might be called brutish, but it worked. $\endgroup$
    – David Cian
    May 18 at 21:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .