12
$\begingroup$

When designing solutions to problems such as the Lunar Lander on OpenAIGym, Reinforcement Learning is a tempting means of giving the agent adequate action control so as to successfully land.

But what are the instances in which control system algorithms, such as PID controllers, would do just an adequate job as, if not better than, Reinforcement Learning?

Questions such as this one do a great job at addressing the theory of this question, but do little to address the practical component.

As an Artificial Intelligence engineer, what elements of a problem domain should suggest to me that a PID controller is insufficient to solve a problem, and a Reinforcement Learning algorithm should instead be used (or vice versa)?

$\endgroup$
  • $\begingroup$ The basic idea I have about PID says its not easy to design. It has lots of integrals and differentials involved. So this is basically the same idea as when you replace statistics with ML approaches. Control systems is definitely flawless but it is too much work. $\endgroup$ – DuttaA May 22 at 17:26
  • 2
    $\begingroup$ actually it's not too much work, its pretty standard in industry, using modern system design tools like MATLAB you can tune PID or any other controller relatively easy to satisfy your needs. Reinforcement learning is not applied in practice since it needs abundance of data and there are no theoretical garanties like there is for classic control theory. By the way, contoller design does not involve working directly with integrals/differentials, for linear systems all the work is done in Laplace domain which involves simple algebraic manipulations $\endgroup$ – Brale_ May 22 at 18:27
  • $\begingroup$ @Brale_ but it still involves a lot of theoretical knowledge..Laplace domain only simplifies the differential but you need to know how to design things (poles and zeros) such that systems don't become unstable. It's pretty tough to visualize to me how those things actually work. $\endgroup$ – DuttaA May 22 at 19:06
  • 1
    $\begingroup$ As a good rule of thumb that helped me in past projects, if you cant explain explain the optimal policy (PID, RL, or otherwise) in a few sentences, PIDs will be really really hard. What's the optimal policy for Pacman? $\endgroup$ – Jaden Travnik May 22 at 22:39
5
$\begingroup$

I think the comments are basically on the right track.

PID controllers are useful for finding optimal policies in continuous dynamical systems, and often these domains are also used as benchmarks for RL, precisely because there is an easily derived optimal policy. However, in practice, you'd obviously prefer a PID controller for any domain in which you can easily design one: the controller's behaviors are well understood, while RL solutions are often difficult to interpret.

Where RL shines is in tasks where we know what good behaviour looks like (i.e., we know the reward function), and we know what sensor inputs look like (i.e. we can completely and accurately describe a given state numerically), but we have little or no idea what we actually want the agent to do to achieve those rewards.

Here's a good example:

  • If I wanted to make an agent to maneuver a plane from in front of an enemy plane with known movement patterns to behind it, using the least amount of fuel, I'd much prefer to use a PID controller.

  • If I wanted to make an agent to control a plane and shoot down an enemy plane with enough fuel left to land, but without a formal description of how the enemy plane might attack (perhaps a human expert will pilot it in simulations against our agent), I'd much prefer RL.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.