2
$\begingroup$

As the question suggests, I'm trying to see if I can solve OpenAI's hardcore version of their gym's bipedal walker using OpenAI's DDPG algorithm.

Below is a performance graph from my latest attempt, including the hyper parameters, along with some other attempts I've made. I realise it has been solved using other custom implementations (also utilising only dense layers in Tensorflow, not convolution), but I don't seem to understand why it seems so difficult to solve using OpenAI's implementation of DDPG? Can anyone please point out where I might be going wrong? Thank you so much for any help!

Latest attempt's performance: DDPG performance - Bipedal Walker Hardcore v2

  • Average score: about -75 to -80
  • Env interacts: about 8.4mil (around 2600 epochs)
  • Batch size: 64
  • Replay memory: 1000000
  • Network: 512, 256 (relu activation on inputs, tanh on outputs)
  • All other inputs left to default

Similar experiments yielded similar scores (or less), and included:

  • Network sizes of (400,300), (256,128), and (128,128,128)
  • Number of epochs ranging from 500 all the way to 100000
  • Replay memory sizes all the way up to 5000000
  • Batch sizes of 32, 64, 128, and 256
  • All of the above, with both DDPG as well as TD3

Thank you so much for any help! It would be greatly appreciated!

$\endgroup$
2
$\begingroup$

You may be very interested to know that there was a bug in the v2 Lidar tracing, making the agent think there were phantom objects, and sometimes intersecting with its own legs:

https://github.com/openai/gym/pull/1789

Finding this bug makes me even more impressed anyone has solved BipedalWalkerHardcore-v2 - it seems the observations from lidar have been inconsistent and incorrect, returning the furthest hit result instead of closest.

...

Before fix - lidar traces through ground, and hits the side of a pit, giving the agent the impression of a "phantom canyon" in front of the pit, that only appears as it approaches the pit: https://i.imgur.com/XKPnRTR.png

...

After fix - lidar is stopped by terrain, even when another object is behind it: https://i.imgur.com/Gg8B5BD.png

...

After triple checking the docs - I've submitted a minor tweak (returning -1 instead of 1 for an object that should be ignored) - it now seems legs are correctly ignored, and the traces are accurate in all situations!

...

It seems to me that solutions to BipedalWalkerHardcore-v2 have not just learned to deal with the complex environment - but advanced a step ahead, and are able to deal with the complex environment and sensory hallucinations causing them to jump at the slightest hint of a cube, and keep running even when it looks like the ground is not visible below their feet, relying more on the touch sensor than the lidar, or perhaps recognising the difference in "shape" between a real pit and a "fake pit" (A real pit has a floor)

BipedalWalkerHardcore-v2 has been bumped to BipedalWalkerHardcore-v3 with these fixes as of Jan 31, 2020.

You might want to try retraining your agent now! (although it is still a difficult environment to solve)

To expand on why DDPG doesn't solve it, when although buggy, BipedalWalkerHardcore-v2 is solvable: The solution landscape to this problem is as full of pits as the environment itself. To learn to leap over a pit in the environment for example, the agent must perform a complex sequence of actions that is difficult to discover by random chance. Each time it fails, it learns that being close to a pit is highly likely to result in a large penalty, and in an effort to maximise it's rewards, will often remain stationary with a naive method like DDPG as the rewards for doing that are higher than trying and falling into the pit once more. In short, vanilla DDPG lacks enough exploration power to find the complex series of actions required before it converges on not going near the pit. Not to mention all of the other things it needs to learn to be successful.

Of the very few published examples that have solved it, one used Evolutionary Strategy - a gradient free method essentially trying millions of policies and evaluating them, and a custom A3C method that was tailored to solve this particular environment. Both had high computational and sample requirements. I speak from experience as I have personally solved this environment with a RL exploration algorithm that generalises to other environments, solves it in 4 hours on a single cpu, and can be used with any off-policy RL algorithm, but unfortunately I'm unable to publish it because of IP with the company I work for.

TLDR; The chance of vanilla DDPG solving it are infeasibly small.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you so much for your answer! It's great to get some succinct and useful feedback from someone who has explored this themselves, and I'm anxious to dive in and try again, especially with the new updates. +1 $\endgroup$ – Chris Kempen Feb 6 at 7:47
  • $\begingroup$ It's pretty impressive that you have an algorithm that solves something like this in 4 hours on a cpu, that must be state of the art for exploration, too bad you can't share it. $\endgroup$ – Brale Feb 6 at 16:16
  • 1
    $\begingroup$ @Brale Thank you, I was pretty happy with the results :) I've seen recent papers by DeepMind and Academics that are starting to look into the same concept, but DeepMind's is for pixel environments and the other was gridworlds. As far as I know, it's a novel implementation for control environments. It's essentially learned exploration instead of random noise... $\endgroup$ – parrowdice Feb 7 at 1:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.