According to the original Proximal Policy Optimization paper (PPO paper), we always sample an action from the actor distribution.

According to the link

The overall loss is calculated as $\text{loss} = pg_\text{loss} - \text{entropy} \times \text{ent}_\text{coef} + vf_\text{loss} \times vf_\text{coef}$, which includes entropy maximization, which intuitively encourages the exploration by encouraging the action probability distribution to be more chaotic.

Why do we explicitly force our agent to over-explore? While I understand why we do that for the first $m$ epochs, I do not see a reason why it is always done instead of choosing the best action (in case of discrete action space) for the last $N-m$ epochs?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.