1
$\begingroup$

Im currently working with A2C. The model was able to learn open ai pong, i ran this as a sanity check that i havent made any bugs. Now im trying to make the model play breakout, but still after 10m steps the model has not made any significant progress. Im using baseline hyperparameters which can be found here https://github.com/openai/baselines/blob/master/baselines/a2c/a2c.py, except my buffersize have been from 512 to 4096. Ive noticed that entropy decreases extremely slowly

enter image description here

given the buffersize from the interval which i just gave. So my questions are how to make entropy decrease and how to increase rewards per buffer? Ive tried to decrease the entropy coefficient to almost zero, but still it acts very weirdly.

Update: Even when i set the entropy coef to zero entropy wont decrease, i guess i might have a bug?

enter image description here

enter image description here

$\endgroup$

1 Answer 1

0
$\begingroup$

Human performance on Breakout is ~30, if you refer to the original DQN paper (Table 1). In the original A3C paper, it takes around 5 epochs to reach that score, so 20 millions frames (Figure 3).

Is the total loss going down in your original experiment? If you set the entropy coefficient to zero, then the contribution of the entropy loss to the overall loss is zero'ed out. That's why it's actually expected that the entropy loss won't decrease.

You're taking the gradients of $$L_{\text{total}}(\theta_{\text{policy}}, \theta_{\text{value}}, \theta_{\text{entropy}}) = L_{\text{policy gradient}}(\theta_{\text{policy}}) + c * L_{\text{value}}(\theta_{\text{value}}) - 0 * L_{\text{entropy}}(\theta_{\text{entropy}}) = L_{\text{policy gradient}}(\theta_{\text{policy}}) + c * L_{\text{value}}(\theta_{\text{value}})$$ where $\theta_{\text{policy}}$ are the parameters of the policy network, etc.

Therefore, $L_{\text{entropy}}(\theta_{\text{entropy}})$ can be anything, it's unconstrained by the optimization of $L_{\text{total}}$

To test that you don't have a bug, you should in fact increase the entropy coefficient and check that the entropy loss is decreasing.

As a reminder, the entropy loss is actually here to maximize the entropy of the policy so that it doesn't collapse to a deterministic one.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .