2
$\begingroup$

I know that classical control systems have been used to solve the problem of the inverted pendulum - inverted pendulum.

But I've seen that people have also used machine learning techniques to solve this nowadays - machine learning of inverted pendulum.

I came across a video on how to apply a machine learning technique called reinforcement learning on openAI gym - OpenAI gym reinforcement learning.

My question is, can I use this simulation and use it to train a controller for a real-world application of inverted pendulum?

$\endgroup$
  • 1
    $\begingroup$ Hi, welcome to AI Stack Exchange. I'm not sure I understand your last sentence - do you mean you want to build a controller (maybe trained using RL) using the simulation, then attempt to use it to interface to and control a physical cart pole system that you have been supplied or intend to build? $\endgroup$ – Neil Slater Nov 18 at 18:57
  • $\begingroup$ Yes,I was thinking I could do that. $\endgroup$ – blazingcannon Nov 19 at 4:31
1
$\begingroup$

In general, you can use a simulation to prepare and train a controller for a real world application. A good example of this being done for robotics is in the paper Autonomous helicopter flight via reinforcement learning where a Reinforcement Learning agent was trained on a model of helicopter dynamics before being used in reality. Often, as in this case, such work is done to avoid expensive failures due to the trial and error nature of RL - if an error is expensive, such as crashing a helicopter, then ideally the agent performs the checks to avoid it in simulation, by planning or some other virtual environment as opposed to in the real world.

The main hurdle to completing training in simulation then transfering to the real world, is the fidelity of the simulation. The simulation's physics, including measurements of physical quantities, the size of time steps, amount of randomness/noise, should match between the simulation and the target real-world environment. If they do not match, then a learning agent could generate a policy that works in simulation, but that fails in reality.

For the autonomous helicopter, the researchers used data from a human operator controlling the real helicopter, to help generate a predictive model that was used in the simulation.

Can you do the same with Open AI Gym environments? Probably not, unfortunately. The main issue is that the units used are fixed in most environments, and are unlikely to closely relate to any specific real world implementation of the same kind of system. In addition, the physics is often simplified - probably a minor issue for CartPole, but a more major one for environments like LunarLander which ignores weight of fuel used and is a 2D simulation of a 3D environment.

So, for instance, in CartPole environments, the following values are fixed:

  • Size of time step
  • Mass of cart
  • Mass and length of pole to be balanced
  • Force that cart motor pushes with

There are a couple of approaches you could use to work around this:

  1. Make a new version of the environment and adjust it so that values match to a real environment you want to train for. Note this may still be limited, as the physics model is still quite simple, and may not allow for the real operating characteristics of the cart motor.

  2. Use the CartPole environment as-is, not to train a controller directly, but to select hyper parameters, such as neural network size, learning rate etc. That will result in a learning agent that you are reasonably confident can learn policies with the state representation and general behaviour of your target system. You then train "for real" again in the physical system.

You can combine these ideas, creating a best-guess controller from simulation, then refining it in a real environment by continuing the training on a real system.

$\endgroup$
  • $\begingroup$ Thanks. Since I am new to machine learning, I don't know much about it. In ordinary control systems that I am familiar with, we use PID to tune the controller. I've seen some videos of an inverted pendulum using reinforcement learning. How does reinforcement learning work? Does it learn to tune the values of PID automatically? Or is it a different algorithm altogether? $\endgroup$ – blazingcannon Nov 19 at 14:31
  • $\begingroup$ @blazingcannon: I cannot answer that in comments with any level of detail, and suggest you ask a new question, but take care not to make it too broad. Reinforcement learning basically uses trial and error to tune a policy which maps observed state to a controler output (an "action"). Although it uses different terminology to classic control, a lot of the core ideas are the same. RL can solve wider class of problems than PID, but at the expense of being less efficient where PID works well. $\endgroup$ – Neil Slater Nov 19 at 16:20
  • $\begingroup$ Thanks, I will consider asking another question. But since you said PID is more efficient than reinforcement learning, I am prompted to ask whether PID can outperform RL in situations where the environment keeps changing? $\endgroup$ – blazingcannon Nov 19 at 16:46
  • $\begingroup$ @blazingcannon: That will all depend on details, such as the nature of the change and the orginal control task. You won't get any PID to play a game of Go at all, whilst the best RL agent (Alpha Zero) should cope just fine if the opponent it faced changed every turn $\endgroup$ – Neil Slater Nov 19 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.