# Is it common to have extreme policy's probabilities?

I have implemented several policy gradient algorithms (REINFORCE, A2C, and PPO) and am finding that the resultant policy's action probability distributions can be rather extreme. As a note, I have based my implementations on OpenAI's baselines. I've been using NNs as the function approximator followed by a Softmax layer. For example, with Cartpole I end up with action distributions like $$[1.0,3e-17]$$. I could understand this for a single action, potentially, but sequential trajectories end up having a probability of 1. I have been calculating the trajectory probability by $$\prod_i \pi(a_i|s_i)$$. Varying the learning rate changes how fast I arrive at this distribution, I have used learning rates of $$[1e-6, 0.1]$$. It seems to me that a trajectory's probability should never be 1.0 or 0.0 consistently, especially with a stochastic start. This also occurs for environments like LunarLander.

For the most part, the resulting policies are near-optimal solutions that pass the criteria for solving the environments set by OpenAI. Some random seeds are sub-optimal

I have been trying to identify a bug in my code, but I'm not sure what bug would be across all 3 algorithms and across environments.

Is it common to have such extreme policy's probabilities? Is there a common way to handle an update so the policy's probabilities do not end up so extreme? Any insight would be greatly appreciated!

• Are the eventual policies near optimal, do they solve the Gym environments? E.g. for LunarLander do you get consistently ~200 reward? Jul 20 '20 at 21:59
• For the most part, yes. Some random seeds are sub-optimal. Jul 21 '20 at 14:10