For reinforcement learning implementations with a gym.spaces.Box action space, which is the product of $k$ real closed intervals, it is common (actually more like ubiquitous) to use a diagonal Gaussian distribution to sample actions (see e.g. https://github.com/openai/baselines/blob/master/baselines/common/distributions.py). This means that, if we are sampling $k$ real-valued continuous subactions, call them $(a_1, \dots, a_k)$, from some random vector $\mathbf{A} = (A_1, \dots, A_k)$, then the $\mathrm{Cov}(A_i, A_j) = 0$ for all $i \neq j$. So they all have zero correlation (though they're not necessarily independent).

Using torch as an example framework, is sampling from torch.distributions.multivariate_normal.MultivariateNormal with a diagonal covariance matrix equivalent to sampling from torch.distributions.normal.Normal with the same mean vector and the diagonal of the covariance matrix as your standard deviation vector? Also, in what situations would it be useful to learn the whole covariance matrix as parameters, allowing nonzero correlation?

There is some mention of this being slightly more computationally intensive in the following answer: How can policy gradients be applied in the case of multiple continuous actions?. But I am curious if there are some problems where it might be necessary.

It also seems to be common practice to learn the $\mathbf{\sigma}$ vector as a parameter with no inputs, instead of having it be a function of the states/observations. Is there any motivation behind one approach or the other?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.