I am currently writing my bachelor thesis, which is an implementation of proximal policy optimization. Sometimes, I hit a wall because of the gaps in my mathematical knowledge. However, implementing the algorithm helped me to understand the math behind the algorithm.
Unfortunately, I still have a question.
When the action space is continuous, I am using the normal distribution (same as in the PPO implementation by Spinning up). In the mentioned implementation, the logarithm of the standard deviation is used initially to give the same probability to all of the possible action, then they use the standard deviation when choosing an action. Why do we use the logarithm? Why not directly use simply the standard deviation?
I know that the logarithm is easier when it comes to the computations, but I can not see the benefits of the logarithm in the Spinning up implementation.
class MLPGaussianActor(Actor): def __init__(self, obs_dim, act_dim, hidden_sizes, activation): super().__init__() log_std = -0.5 * np.ones(act_dim, dtype=np.float32) self.log_std = torch.nn.Parameter(torch.as_tensor(log_std)) self.mu_net = mlp([obs_dim] + list(hidden_sizes) + [act_dim], activation) def _distribution(self, obs): mu = self.mu_net(obs) std = torch.exp(self.log_std) return Normal(mu, std) def _log_prob_from_distribution(self, pi, act): return pi.log_prob(act).sum(axis=-1) # Last axis sum needed for Torch Normal distribution