Is (log-)standard deviation learned in TRPO and PPO or fixed instead?

Question

After having read Williams (1992), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating continuous output values, I assumed that this would be common practice nowadays in the domain of Deep Reinforcement Learning (DRL). In the supplementary material associated with the paper introducing Trust Region Policy Optimization (TRPO), however, it is stated that:

A neural network with several fully-connected (dense) layers maps from the input features to the mean of a Gaussian distribution. A separate set of parameters specifies the log standard deviation of each element. More concretely, the parameters include a set of weights and biases for the neural network computing the mean, $\{W_i , b_i\}_{i=1}^L$ , and a vector $r$ (log standard deviation) with the same dimension as $a$. Then, the policy is defined by the normal distribution $\mathcal{N}(\text{mean}=\text{NeuralNet}(s; \{W_i , b_i\}_{i=1}^L), \text{stdev}=\text{exp}(r))$.

where $s$ refers to a state and $a$ to a predicted action (respectively a vector of actions if multiple outputs are generated concurrently).

To me this suggests that the standard deviation stdev (being a function of $r$) is actually not learned when training a TRPO agent, but that it is solely determined by some possibly constant vector $r$.

Since I found the idea of adjusting both the mean and standard deviation together when training a REINFORCE agent quite reasonable, I got wondering whether it is actually true that TRPO agents do not treat the standard deviation for sampling output values as a trainable parameter, but just as a function of the state-independent vector $r$. (Pretty much the same shall then apply to Proximal Policy Optimization (PPO) agents as well, since they are reported to follow TRPO's model architecture in the continuous output case.)

In search for an answer, I browsed OpenAI's baselines repository containing reference implementations of both TRPO and PPO. In my understanding of their code, the code seems to confirm my assumption that standard deviation is a non-trainable parameter and that it is, instead of being trainable, taken to be a constant.

Now, I was wondering whether my understanding of the procedure how TRPO (and PPO) computes standard deviation(s) is correct or whether I misunderstood or overlooked something important here.

Answer 1

Using PPO from stable baselines 3, I see clearly that the variance of the actions are reducing in the environment I am using, where the optimal actions are static and not dependent on the state. This seems to suggest that the log_std parameter is being learned for the SBL3 implementation.

In the docs for the SBL3-PPO, the description says log_std_init is the initial value which also suggests it is a learned parameter.

log_std_init (float) – Initial value for the log standard deviation

Of course, it is ultimately a design choice whether you want to make this parameter learnable.

Update: Upon inspecting the PPO paper again, it is stated that they using linear annealing of the log_std parameter. However, when I inspect the log_std parameter in my experiments which start at 0 and has 3 dimensions, the values are not the same. This leads me to think there is both annealing and learning at the same time.