# How to tune hypeparametes in A2C-ppo?

Im currently working with A2C. The model was able to learn open ai pong, i ran this as a sanity check that i havent made any bugs. Now im trying to make the model play breakout, but still after 10m steps the model has not made any significant progress. Im using baseline hyperparameters which can be found here https://github.com/openai/baselines/blob/master/baselines/a2c/a2c.py, except my buffersize have been from 512 to 4096. Ive noticed that entropy decreases extremely slowly

given the buffersize from the interval which i just gave. So my questions are how to make entropy decrease and how to increase rewards per buffer? Ive tried to decrease the entropy coefficient to almost zero, but still it acts very weirdly.

Update: Even when i set the entropy coef to zero entropy wont decrease, i guess i might have a bug?

You're taking the gradients of $$L_{\text{total}}(\theta_{\text{policy}}, \theta_{\text{value}}, \theta_{\text{entropy}}) = L_{\text{policy gradient}}(\theta_{\text{policy}}) + c * L_{\text{value}}(\theta_{\text{value}}) - 0 * L_{\text{entropy}}(\theta_{\text{entropy}}) = L_{\text{policy gradient}}(\theta_{\text{policy}}) + c * L_{\text{value}}(\theta_{\text{value}})$$ where $$\theta_{\text{policy}}$$ are the parameters of the policy network, etc.
Therefore, $$L_{\text{entropy}}(\theta_{\text{entropy}})$$ can be anything, it's unconstrained by the optimization of $$L_{\text{total}}$$