Why is the logarithm of the standard deviation used in this implementation of proximal policy optimization?

Question

I am currently writing my bachelor thesis, which is an implementation of proximal policy optimization. Sometimes, I hit a wall because of the gaps in my mathematical knowledge. However, implementing the algorithm helped me to understand the math behind the algorithm.

Unfortunately, I still have a question.

When the action space is continuous, I am using the normal distribution (same as in the PPO implementation by Spinning up). In the mentioned implementation, the logarithm of the standard deviation is used initially to give the same probability to all of the possible action, then they use the standard deviation when choosing an action. Why do we use the logarithm? Why not directly use simply the standard deviation?

I know that the logarithm is easier when it comes to the computations, but I can not see the benefits of the logarithm in the Spinning up implementation.

class MLPGaussianActor(Actor):

    def __init__(self, obs_dim, act_dim, hidden_sizes, activation):
        super().__init__()
        log_std = -0.5 * np.ones(act_dim, dtype=np.float32)
        self.log_std = torch.nn.Parameter(torch.as_tensor(log_std))
        self.mu_net = mlp([obs_dim] + list(hidden_sizes) + [act_dim], activation)

    def _distribution(self, obs):
        mu = self.mu_net(obs)
        std = torch.exp(self.log_std)
        return Normal(mu, std)

    def _log_prob_from_distribution(self, pi, act):
        return pi.log_prob(act).sum(axis=-1)    # Last axis sum needed for Torch Normal distribution

Answer 1

As you have mentioned using log is nicer because it makes multiplications to additions etc etc(it helps in numerical stability issues). But I think over here, the reason they are doing it like that is because of enforcing a simple constraint in a much more simpler way. In the __init__ we are noticing that the log_std is being formulated instead of the std itself. Would it be wrong if you formulated the std itself? No. But it would be a bit messy to have some constrained imposed on it. For example, if we modeled std it could have been possible for the std to become 0 or negative. Well that's not a correct values of a std. But over here that is being enforced automatically, this kind of little things makes the learning for the model easier. Even if the log_std becomes negative or zero it does not matter because the exp will take care of it by exponentiating it to a positive number.