1
$\begingroup$

I am currently trying to train BipedalWalker of OpenAI gym by using policy gradient approach. My action space contains 4 continuous actions, all ranging [-1.0, 1.0]. In this case, how can we calculate gradient of log policy when log(negative_action) gives nan value.

$\endgroup$
11
  • 1
    $\begingroup$ You don’t want to take log of the action. You want the log likelihood of taking an action. If your network outputs parameters of a Gaussian distribution then you would want to evaluate $\log f(a_t| \mu, \sigma)$, where $f$ is the pdf of the standard normal. Note that a tanh activation is usually wrapped around the action when the action space is symmetric in these MuJoCo environments and so the gradient will change slightly as your distribution is now $Y = \text{tanh}(X)$. $\endgroup$
    – David
    Mar 5, 2022 at 9:53
  • $\begingroup$ Correct me if I am wrong. Mean, SD = Policy_Model(current_state). Then, action = random.gauss(mean, SD). Then probability_density_function = exp(-0.5((x-mu)/SD)^2)/(SD√2pi). Then, log of this function then gradient. Also, I didn't get why we use tanh in the policy_models last layer. $\endgroup$
    – Himanshu
    Mar 6, 2022 at 11:25
  • $\begingroup$ Why are we using tanh for getting mu and sd. $\endgroup$
    – Himanshu
    Mar 6, 2022 at 11:26
  • $\begingroup$ You are correct that action comes from sampling the Gaussian. However, the support of the Gaussian random variable is $\mathbb{R}$, so your action can be any real number. As you want it to be constrained to $[-1, 1]$ you apply the tanh function to enforce this. $\endgroup$
    – David
    Mar 6, 2022 at 13:33
  • $\begingroup$ Am I correct about the probability density function. Also, should I use separate gaussian parameters for all four i.e., [mu1, mu2, mu3, mu4, sig1, sig2, sig3, sig4] = policy_model(current_state) Or should I use multivariate guassian distribution which has a covariance matrix. $\endgroup$
    – Himanshu
    Mar 6, 2022 at 13:37

1 Answer 1

1
$\begingroup$

The policy gradient tells us that $$\nabla_\theta v_\pi(s) = \mathbb{E}_\pi\left[ G_t \nabla_\theta \log \pi_\theta(a | s) \right] \; ;$$ where as usual $G_t$ are the returns, $\pi$ is policy parameterised by $\theta$ and the expectation is taken with respect to the policy $\pi$ and the state distribution induced by $\pi$.

Hopefully you can see that here we do not need to take the log of the actions, but instead the log of the probability of taking an action (or, in a continuous action case, the log of the pdf of taking an action), i.e. the log-likelihood of an action.

In your case, as with most MuJoCo environments, it is common to train a neural network to output the mean and standard deviation of a Gaussian distribution -- assuming the action has $d$ dimensions, this is done by having a neural network with two heads, i.e. a neural network that takes the input and outputs two $d$ length vectors. Note that in practice, it is common for the network to output the log-standard deviation for stability reasons. Further, it is typically assumed that the log-sd is clamped to some pre-specified range too, again to help with numerical stability. For $d > 1$ we assume that the Multi-Variate Gaussian has diagonal covariance matrix, so the neural network still need only predict a $d$ length vector for the log-sd, as opposed to a full covariance matrix.

Now, as the action range for the MuJoCo environments is typically symmetric, it is also common to see that a tanh activation is applied to the action to get the action to be in the range (-1, 1). This makes things a bit more nuanced in terms of how to apply the policy gradient, because your random variable (the action) is no longer gaussian. Luckily, as we have assumed independent gaussians, the resulting distribution of the random variable is not that tricky to calculate analytically.

If we assume that $A$ is the random variable representing the action, then $A$ is defined as $A = \tanh(X)$, where $X \sim \mbox{MVN}(\boldsymbol{\mu}_\theta, \boldsymbol{\Sigma}_\theta)$, where $\boldsymbol{\mu}, \boldsymbol{\Sigma}$ are the predicted mean and (diagonal) co-variance parameters of the gaussian, with the $\theta$ subscript added to remind us of the fact that they are predicted by the NN parameterised by $\theta$. Now, we can use the fact that the pdf of an arbitrary (one-to-one) function of a random variable defined by $Y = g(X)$ is given (for the univariate case) by $$f_Y(y) = f_X\left(g^{-1}(y)\right) \left|\frac{dx}{dy}\right| \; ;$$ note that the multivariate case is a simple extension of this and details can be found online.

In our example, $g = \tanh$, and so the pdf of the policy is given by $$\pi_\theta(\textbf{a}|s) = \phi_\theta(\textbf{x}|s)\left|\mbox{det}\left(\frac{d\textbf{x}}{d\textbf{a}}\right)\right| = \phi_\theta(\textbf{x}|s)\left|\mbox{det}\left(\frac{d\textbf{a}}{d\textbf{x}}\right)\right|^{-1} \; ;$$ where $\textbf{x} \sim X$ and $\phi$ is the pdf of the MVN with $\theta$ notation to imply that the mean and covariance come from the NN prediction. Note that we invert the determinant of the transformation because it is often easier to calculate the derivative this way; in the case of $tanh$ being the transformation function along with the fact that we assume the MVN has diagonal covariance matrix we get that $\frac{d\textbf{a}}{d\textbf{x}} = \mbox{diag}(1 - \tanh^2(\textbf{x}))$. Now, using this, you can evaluate $\log \pi_\theta(a|s)$ in the policy gradient term correctly by using the correct pdf.

$\endgroup$
3
  • $\begingroup$ I understood up to the 5th para. I couldn't understand what is fy = fx(g(x)|dx/dy|. Can you please elaborate more or suggest me some content to understand that? $\endgroup$
    – Himanshu
    Mar 10, 2022 at 5:56
  • $\begingroup$ If you go to the wiki page for a random variable and click the section ‘function of a random variable’ it should explain what this paragraph means, and also has some examples you can work through. $\endgroup$
    – David
    Mar 10, 2022 at 10:01
  • $\begingroup$ I have updated my question and included the code. Please help me find the reason why my agent is note converging. $\endgroup$
    – Himanshu
    Mar 11, 2022 at 7:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .