After having read Williams (1992), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating continuous output values, I assumed that this would be common practice nowadays in the domain of Deep Reinforcement Learning (DRL). In the supplementary material associated with the paper introducing Trust Region Policy Optimization (TRPO), however, it is stated that:

A neural network with several fully-connected (dense) layers maps from the input features to the mean of a Gaussian distribution. A separate set of parameters specifies the log standard deviation of each element. More concretely, the parameters include a set of weights and biases for the neural network computing the mean, $\{W_i , b_i\}_{i=1}^L$ , and a vector $r$ (log standard deviation) with the same dimension as $a$. Then, the policy is defined by the normal distribution $\mathcal{N}(\text{mean}=\text{NeuralNet}(s; \{W_i , b_i\}_{i=1}^L), \text{stdev}=\text{exp}(r))$.

where $s$ refers to a state and $a$ to a predicted action (respectively a vector of actions if multiple outputs are generated concurrently).

To me this suggests that the standard deviation stdev (being a function of $r$) is actually not learned when training a TRPO agent, but that it is solely determined by some possibly constant vector $r$.

Since I found the idea of adjusting both the mean and standard deviation together when training a REINFORCE agent quite reasonable, I got wondering whether it is actually true that TRPO agents do not treat the standard deviation for sampling output values as a trainable parameter, but just as a function of the state-independent vector $r$. (Pretty much the same shall then apply to Proximal Policy Optimization (PPO) agents as well, since they are reported to follow TRPO's model architecture in the continuous output case.)

In search for an answer, I browsed OpenAI's baselines repository containing reference implementations of both TRPO and PPO. In my understanding of their code, the code seems to confirm my assumption that standard deviation is a non-trainable parameter and that it is, instead of being trainable, taken to be a constant.

Now, I was wondering whether my understanding of the procedure how TRPO (and PPO) computes standard deviation(s) is correct or whether I misunderstood or overlooked something important here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.