# What is the loss for policy gradients with continuous actions?

I know with policy gradients used in an environment with a discrete action space are updated with $$\Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}$$ where v could be many things that represent how good the action was. And I know that this can be calculated by performing cross entropy loss with the target being what the network would have outputted if it were completely confident in its action (zeros with the index of the action chosen being one). But I don’t understand how to apply that to policy gradients that output the mean and variance of a Gaussian distribution for a continuous action space. What is the loss for these types of policy gradients?

I tried keeping the variance constant and updating the output with mean squared error loss and the target being the action it took. I thought this would end up pushing the mean towards actions with greater total rewards but it got nowhere in OpenAI’s Pendulum environment.

It would also be very helpful if it was described in a way with a loss function and a target, like how policy gradients with discrete action spaces can be updated with cross entropy loss. That is how I understand it best but it is okay if that is not possible.

Edit: My implementation with PyTorch is not working for the pendulum enviornment. I have tried changing the learning rate and sigma, using actor critic methods,having the network output the mean and variance, and training for thousands of episodes with different batch sizes. I used an actor critic method that worked with CartPole and Lunar Lander then changed a couple of lines including the distribution from categorical to normal but the agent never learned. Here is a reproducible example:


import torch
import torch.nn as nn
import torch.optim as optim
from torch.distributions.normal import Normal
import numpy as np
import gym
import matplotlib.pyplot as plt

class Agent(nn.Module):
def __init__(self,lr):
super(Agent,self).__init__()
self.fc1 = nn.Linear(3,64)
self.fc2 = nn.Linear(64,32)
self.fc3 = nn.Linear(32,1) #neural network with layers 3,64,32,1

def forward(self,x):
x = torch.relu(self.fc1(x)) #relu and tanh for output
x = torch.relu(self.fc2(x))
x = torch.tanh(self.fc3(x))
return x

env = gym.make('Pendulum-v0')
agent = Agent(0.0001) #hyperparameters
SIGMA = 0.15
DISCOUNT = 0.99
total = []

for e in range(500):
log_probs, rewards = [], []
done = False
state = env.reset()
while not done:
mu = agent(torch.from_numpy(state).float()) #mean of gaussian distribution
distribution = Normal(mu*2,0.15) #create distribution with constant sigma and mean multiplied by 2
action = distribution.sample() #randomly sample from distribution
state,reward,done,info = env.step([action])
log_probs.append(distribution.log_prob(action)) #log prob of action
rewards.append(reward)
total.append(sum(rewards))

cumulative = 0
d_rewards = np.zeros(len(rewards))
for t in reversed(range(len(rewards))): #get discounted rewards
cumulative = cumulative * DISCOUNT + rewards[t]
d_rewards[t] = cumulative
d_rewards -= np.mean(d_rewards) #normalize
d_rewards /= np.std(d_rewards)

loss = 0
for t in range(len(rewards)):
loss += -log_probs[t] * d_rewards[t] #loss is - log prob * total reward

loss.backward() #update
agent.optimizer.step()

if e%10==0:
print(e,sum(rewards))
plt.plot(total,color='blue') #plot
plt.pause(0.0001)

def run(i): #to visualize performance
for _ in range(i):
done = False
state = env.reset()
while not done:
env.render()
mu = agent(torch.from_numpy(state).float())
distribution = Normal(mu*2,0.15)
action = distribution.sample()
state,reward,done,info = env.step([action])
env.close()

$$$$

• You might want to first express your policy as a density function over the space of possible actions. If you have one of these for each state, and the states are a continuous space as well, you might want to look at Gaussian processes and the likes. – Robby Goetschalckx Sep 30 '20 at 22:36
• @Robby I don’t understand exactly what you mean. Are you saying I should make the continuous action space discrete? Like if the range of possible actions is -1 to 1 then have '[-1,-0.5,0,0.5,1]' as the only possible actions? I already know how to do that, I am looking for a way to do it with continuous action spaces for problems where that would not work. – S2673 Oct 1 '20 at 1:34
• No, you don't want it to be discrete. A policy over a continuous space is necessarily a probability density function. With enough training, this should converge to a Dirac delta function centered on the optimal action. If you parameterize a density, you can use that in the formula you have, and update the parameters according to the gradient. – Robby Goetschalckx Oct 1 '20 at 5:49
• @RobbyGoetschalckx I think worth puting in an answer. Probably the most usual way to do this with is have the NN output params of a gaussian - mean and sd (or variance). Showing the gradient terms for mean and sd output when choosing action=x from that distribution should answer OP's question. – Neil Slater Oct 1 '20 at 6:49
• @Robby The goal is for it to converge to a Dirac delta function. But what does that update look like with the network outputting the mean and variance of a Gaussian distribution? I am using the Gaussian distribution because it seems like the most common one and after a quick google search I couldn’t find an example of policy gradients with a different distribution function. – S2673 Oct 1 '20 at 11:39

This update rule can still be applied in the continuous domain.

As pointed out in the comments, suppose we are parameterising our policy using a Gaussian distribution, where our neural networks take as input the state we are in and output the parameters of a Gaussian distribution, the mean and the standard deviation which we will denote as $$\mu(s, \theta)$$ and $$\sigma(s, \theta)$$ where $$s$$ shows the dependancy of the state and $$\theta$$ are the parameters of our network.

I will assume a one-dimensional case for ease of notation but this can be extended to multi-variate cases. Our policy is now defined as $$\pi(a_t | s_t) = \frac{1}{\sqrt{2\pi \sigma(s_t, \theta)^2}} \exp\left(-\frac{1}{2}\left(\frac{a_t - \mu(s_t, \theta)}{\sigma(s_t, \theta)}\right)^2\right).$$

As you can see, we can easily take the logarithm of this and find the derivative with respect to $$\theta$$, and so nothing changes and the loss you use is the same. You simply evaluate the derivative of the log of your policy with respect to the network parameters, multiply by $$v_t$$ and $$\alpha$$ and take a gradient step in this direction.

To implement this (as I'm assuming you don't want to calculate the NN derivatives by hand) then you could do something along the lines of the following in Pytorch.

First you want to pass your state through your NN to get the mean and standard deviation of the Gaussian distribution. Then you want to simulate $$z \sim N(0,1)$$ and calculate $$a = \mu(s,\theta) + \sigma(s, \theta) \times z$$ so that $$a \sim N( \mu(s, \theta), \sigma(s, \theta))$$ -- this is the reparameterisation trick that makes backpropagation through the network easier as it takes the randomness from a source that doesn't depend on the parameters of the network. $$a$$ is your action that you will execute in your environment and use to calculate the gradient by simply writing the code torch.log(normal_pdf(a, \mu(s, \theta), \sigma(s, \theta)).backward() -- here normal_pdf()` is any function in Python that calculates the pdf of a normal distribution for a given point and parameters.

• Comments are not for extended discussion; this conversation has been moved to chat. – nbro Jan 9 at 22:25