# How to reduce the variance of stochastic policy gradient for continuous actions in a partially observable environment?

I am trying to implement a stochastic policy gradient for continuous actions in a partially observable CartPole environment. Specifically, only the current cart position and pole angle are visible, the velocities are invisible. The actions are also continuous ranging from $$-inf$$ to $$inf$$ (the limit of [-1,1] is removed). The problem is considered solve if the cart position and pole angle do not go beyond limit in 1000 steps. The reward for each step is $$1.0-abs(CartPosition/2.4)-abs(PoleAngle/0.25)$$. It is designed to promote the CartPole to stabilize at the field center.

The reinforcement learning algorithm that I am using is REINFORCE with baseline. Other algorithms, like A2C and TD3, are also currently under work, so this question is specifically about REINFORCE with baseline. The discount factor $$\gamma=0.99$$. The network that I am using looks like this (GRU is the Gated Recurrent Unit):

                       |----------------> Linear -> ReLU ------> Linear -> tanh ------> action mean
|                  (in=64,out=64)         (in=64, out=1)
state ----> GRU (2 layers, 64 hidden)--> Linear -> ReLU ------> Linear->Softplus+0.1 ---> action variance
(2-dim)                |                  (in=64,out=64)         (in=64, out=1)
| ---------------> Linear -> ReLU ------> Linear --------------> state value baseline
(in=64,out=64)         (in=64, out=1)


So the network is learning both action mean and action variance, and the action is sampled from a Gaussian distribution with the output mean and variance.

The loss function for state-value network is the "SmoothL1Loss" between the return $$u_t$$ at each step (the discounted sum of reward in the future) and the estimated state-value $$v(s_t)$$ in an episode. $$Loss(ValueNet)=mean(SmoothL1Loss(u_t-v(s_t)))$$

The loss function for the policy network (action mean and variance output) is: $$Loss(PloicyNet)=-mean(\gamma^{t-1}*(u_t-v(s_t))*log(p(a_t)))$$

where $$a_t$$ is the probability density of the action at each time step $$t$$.

The loss for back propogation is $$Loss=Loss(PolicyNet)+Loss(ValueNet)$$. The optimizer is Adam with learning rate set to 0.001. The network is updated after every episode.

My problem is that the network is VERY unstable (though it does solve the task). An example training image is shown below (the orange curve is the smoothed version of the blue curve). I have checked the training logs and find that the action variance drops to about 0.1 after about 10000 episodes, and never rise again. Note that I have manually add 0.1 to action variance, so the output of Softplus function is actually very close to 0. In fact, if the 0.1 is not added, the gradients can become $$NaN$$ because action variance is in the denominator when calculating the gradients (see page 336 at Chapter 13: Policy Gradient Methods in Sutton & Barto's Book).

My questions are:

1. Why is the learning very unstable, and what can be done to improve it?

2. How to properly deal with the variance reducing to zero problem? I don't think that manually adding a constant is a good solution. An article in 1992 actually said scaling the optmization learning rate by the variance at each step of episode can solve the problem, but how to correctly implement this?