0
$\begingroup$

I haven't been able to find any assistance / examples which could help me implement OpenAI's Spinning Up resource to solve Atari's Breakout-v0 game in the OpenAI gym.

I simply want to know why the following command doesn't run, and instead produces an error that I can't find any help on:

python -m spinup.run ppo --env Breakout-v0 --exp_name simpletest

...and then the error:

ValueError: Shape must be rank 2 but is rank 4 for 'pi/multinomial/Multinomial'
 (op: 'Multinomial') with input shapes: [?,210,160,4], [].

I understand the shape dynamics, and have written several (albeit quite unoptimized!) reinforcement learning neural nets in Python, but I was looking forward to using OpenAI's Spinning Up environment to use something more sophisticated and optimized.

Thank you so much for any help on the seeminly noobish question!

$\endgroup$
1
$\begingroup$

So for anyone struggling to understand the OpenAI's Spinning Up educational resource, I'll provide the answer to my question here.

Firstly, it's important to understand that the algorithms expect a 2-dimensional input shape, in rudimentary terms a shape of Box(int), which isn't the case with the default Breakout-v0 game environment, which supplies inputs in the shape Box(210, 160, 3), which is the game screen's height, width, and RGB colour space.

The Breakout-ram-v0 game environment, however, DOES provide an input of the appropriate shape, consequently Box(128,). So switching to this environment solves the initial problem I was having. To train any of the on-policy algorithms (PPO in my case), the command should be:

python -m spinup.run ppo --env Breakout-ram-v0 --exp_name simpletest

To take this a step further, how would you train an off-policy algorithm (such as DDPQ)? Well, the off-policy algorithms expect a continuous action space, which can be seen in the differences in the source code here between MountainCar-v0, and MountainCarContinuous-v0:

self.action_space = spaces.Discrete(3)

...versus:

self.action_space = spaces.Box(low=self.min_action, high=self.max_action,
    shape=(1,), dtype=np.float32)

So, to sum up, how would you train something like the Lunar Lander game environment with the DDPQ algorithm? Well, you'd need the continuous action space version, and by providing it with 2 dense layers with 192 nodes each, I was able to achieve the following results with just 200 epochs using the following command:

python -m spinup.run ddpg --env LunarLanderContinuous-v2 --exp_name ddpq-test --hid [192,192] --epochs 200

LunarLanderContinuous-v2 OpenAI DDPQ test

Pretty cool...anyways, I hope this helps someone sometime, and I would thoroughly recommend looking at the source code for the various OpenAI game environments, as well as their table of game environments for a quick reference to the observation and action spaces. Good luck!

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.