1
$\begingroup$

I have a question about implementing policy gradient methods for problems with continuous action spaces.

Assume that actions are sampled from a diagonal Gaussian distribution with mean vector $\mu$ and standard deviation vector $\sigma$. As far as I understand, we can define a neural network that takes the current state as the input and returns a $\mu$ as its output. According to OpenAI Spinning Up, the standard deviation $\sigma$ can be represented in two different ways:

enter image description here

I don't completely understand the first method. Does it mean that we must set the log standard deviations to fix numbers? Then how do we choose these numbers?

$\endgroup$
1
  • $\begingroup$ If you don't get an answer meanwhile, it may be a good idea to look at existing implementations (e.g. on Github). $\endgroup$
    – nbro
    Jul 19, 2020 at 12:36

1 Answer 1

0
$\begingroup$

The first way, as OpenAI researchers suggest also, is to use a fixed standard deviation to produce continuous actions instead of a trainable one. This value is choosen experimentally (from my experience I recommend between 1 - 1.5, but depending on your problem, other values might fit better). Neglect the 'log', just a positive value is needed (the reason there is a log is because when working with trainable std dev, the network's head produces negative values so they are considered to be log(sigma), they are exponentiated afterwards to obtain sigma and used).

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .