I know with policy gradients used in an environment with a discrete action space are updated with $$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t} $$ where v could be many things that represent how good the action was. And I know that this can be calculated by performing cross entropy loss with the target being what the network would have outputted if it were completely confident in its action (zeros with the index of the action chosen being one). But I don’t understand how to apply that to policy gradients that output the mean and variance of a Gaussian distribution for a continuous action space. What is the loss for these types of policy gradients?
I tried keeping the variance constant and updating the output with mean squared error loss and the target being the action it took. I thought this would end up pushing the mean towards actions with greater total rewards but it got nowhere in OpenAI’s Pendulum environment.
It would also be very helpful if it was described in a way with a loss function and a target, like how policy gradients with discrete action spaces can be updated with cross entropy loss. That is how I understand it best but it is okay if that is not possible.
Edit: My implementation with PyTorch is not working for the pendulum enviornment. I have tried changing the learning rate and sigma, using actor critic methods,having the network output the mean and variance, and training for thousands of episodes with different batch sizes. I used an actor critic method that worked with CartPole and Lunar Lander then changed a couple of lines including the distribution from categorical to normal but the agent never learned. Here is a reproducible example:
import torch
import torch.nn as nn
import torch.optim as optim
from torch.distributions.normal import Normal
import numpy as np
import gym
import matplotlib.pyplot as plt
class Agent(nn.Module):
def __init__(self,lr):
super(Agent,self).__init__()
self.fc1 = nn.Linear(3,64)
self.fc2 = nn.Linear(64,32)
self.fc3 = nn.Linear(32,1) #neural network with layers 3,64,32,1
self.optimizer = optim.Adam(self.parameters(),lr=lr)
def forward(self,x):
x = torch.relu(self.fc1(x)) #relu and tanh for output
x = torch.relu(self.fc2(x))
x = torch.tanh(self.fc3(x))
return x
env = gym.make('Pendulum-v0')
agent = Agent(0.0001) #hyperparameters
SIGMA = 0.15
DISCOUNT = 0.99
total = []
for e in range(500):
log_probs, rewards = [], []
done = False
state = env.reset()
while not done:
mu = agent(torch.from_numpy(state).float()) #mean of gaussian distribution
distribution = Normal(mu*2,0.15) #create distribution with constant sigma and mean multiplied by 2
action = distribution.sample() #randomly sample from distribution
state,reward,done,info = env.step([action])
log_probs.append(distribution.log_prob(action)) #log prob of action
rewards.append(reward)
total.append(sum(rewards))
cumulative = 0
d_rewards = np.zeros(len(rewards))
for t in reversed(range(len(rewards))): #get discounted rewards
cumulative = cumulative * DISCOUNT + rewards[t]
d_rewards[t] = cumulative
d_rewards -= np.mean(d_rewards) #normalize
d_rewards /= np.std(d_rewards)
loss = 0
for t in range(len(rewards)):
loss += -log_probs[t] * d_rewards[t] #loss is - log prob * total reward
agent.optimizer.zero_grad()
loss.backward() #update
agent.optimizer.step()
if e%10==0:
print(e,sum(rewards))
plt.plot(total,color='blue') #plot
plt.pause(0.0001)
def run(i): #to visualize performance
for _ in range(i):
done = False
state = env.reset()
while not done:
env.render()
mu = agent(torch.from_numpy(state).float())
distribution = Normal(mu*2,0.15)
action = distribution.sample()
state,reward,done,info = env.step([action])
env.close()
```
