1
$\begingroup$

There are a lot of examples of balancing a pole (see image below) using reinforcement learning, but I find that almost all examples start close to the upright position.

Is there any good source (or paper) for when the pole actually starts all the way at the bottom?

enter image description here

$\endgroup$
2
$\begingroup$

It is difficult to prove a negative, but I doubt there will be a paper on that specific problem. It should be relatively easy to adjust the environment or write a new one that does this if you wished though.

A very similar environment that does have a lot more written about it is Acrobot, which does have a OpenAI Gym version. Instead of a cart on a track with forces applied and a free join to the pole, there is a longer pole fixed to a free joint, with an active joint in the middle (that the agent can apply forces to). It can be thought of as a very simple model of an acrobat on a trapeze swing (with poles instead of chains, so the swing is stable balanced upside-down)

The degrees of freedom and difficulty of the Acrobot task are similar to CartPole - I would rate it as a harder problem overall, but if you started CartPole with the pole hanging down in fact the two problems are very similar. Usually the joint motor is made too weak to achieve the goal of balancing in a single clean action, and the agent must learn to build momentum over a few swings, before moving to the balance point. That also makes both your alternate start position CartPole and Acrobot similar to MountainCar.

For a low-effort look at a very similar problem you could try Acrobot. Otherwise you will likely need to do some custom work on CartPole.

| improve this answer | |
$\endgroup$
2
$\begingroup$

Almost certainly, there is no such paper since that would be a trivial problem. The pole lying flat is the definition of failure, hence game over. If you started in that position, you would be permanently in the game-over state and you would never learn anything.

The reason is that if the pole is lying flat, then, if you apply a force on the cart (in the same direction as the pole is pointing, say), the pole head moves in exactly the same direction as the cart (i.e. the directional vectors of the cart and pole head movements are identical). Hence, the pole head never moves upwards.

In fact, I am fairly certain, that below certain angle with the surface, the pole can no longer be stabilized. This should be possible to prove from the dynamic equations governing the movement of the cart and the pole. This may not be easy, though, and definitely not easy for me (these are second-order differential equations). Anyway, with this in mind, you can see why one needs to start close enough to the stationary point for the problem to have a solution.

If the pole could go below the surface and swing, however, as here, it would be similar to the acrobot problem mentioned by Neil and you could start anywhere.

| improve this answer | |
$\endgroup$
  • $\begingroup$ "Almost certainly, there is no such paper since that would be a trivial problem." it might seem trivial after it has been solved. I have in the meantime found several papers: scholar.google.com/…. There is also no need to start near the top, the bottom is fine. But yes, it is similar to the acrobat problem. $\endgroup$ – Thomas W May 29 at 7:15
  • $\begingroup$ @ThomasW Trivial in mathematical sense, that was not a reference to anyone's intelligence. If you know that the pole does not move when you apply the force, then the solution is trivial (i.e. any action yields the same outcome). $\endgroup$ – PetrH May 29 at 16:54
  • $\begingroup$ @ThomasW The references you point to analyze the problem, where the pendulum can rotate 360 degrees. That I mentioned in my last sentence as a side note. My answer was about the pole-balancing problem where the pole can only deviate from the upright position by 90 degrees on either side, since this is the problem which you asked about. $\endgroup$ – PetrH May 29 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.