Proximal Policy Optimization for continuous control problem

Question

I am using clipped PPO to train a neural network to act as the controller for steering an aircraft, and am finding that my networks aren't learning. The goal is to keep the aircraft flying to cover the distance, and the code is implemented using Pytorch. I am wondering if anyone who is more experienced would be willing to take a look at my implementation.

Aircraft and actor model
I have a flight simulation environment with the dynamical system of an aircraft, where the simulation is propagated through time in 0.1s increments. The states of the aircraft serve as inputs to the actor network (airspeed, pitch, heading, height), and the controls include pitch and roll angles.

The actor network uses a multivariate normal distribution to output two values that serve as the means for the pitch and roll angle controls. The variances are fixed. This way, the policy is stochastic and allows for exploration. I am using two hidden layers of 256 neurons each for both actor and critic networks, with learning rates of 1e-4. ReLu activation on the hidden layers and sigmoid on the actor's output, with the critic's output being linear.

Reward structure
I have tried two different reward structures:

1. A reward of 1 for every time step that the aircraft is in the air (with a reward of 0 in the terminal state, which is when the aircraft crashes to the ground or if it achieves the maximum flight time)
   
2. A reward of 0 for every time step, but a single reward computed at the terminal state that is proportional to the displacement of the aircraft over the simulation, and an extremely large penalty that is subtracted from this reward if the aircraft crashes.

Neither of these reward structures seemed to allow learning. I am wondering if simply having a reward of 1 at every time step, much like the gym cartpole environment is insufficient in getting the model to learn how to fly. I was expecting the network to figure out ways of controlling the aircraft to fly longer to obtain a greater total reward. For the second reward structure, I expected the advantage estimation to carry the large reward/penalty backward through the trajectory so that the actor would learn to avoid flying in certain ways that end up in crashing the plane.

PPO implementation
For the clipped PPO implementation, I am:

1. Resetting the flight simulation environment
   
2. Propagating the simulation until the aircraft crashes or reaches maximum flight time
   
3. Training the actor and critic networks every 20 time steps (2s of simulation)
   
4. Repeating this for a maximum number of simulations

For training, I take the 20 time steps, shuffle them, and divide them into 5 batches. Then I train the network on these batches, reshuffle the 20 time steps and create 5 new batches, and train again. I repeat this process for a total of 4 epochs per learning cycle.

The fastest that the aircraft can reach the ground (crash) is 1.7s of simulation time (17 time steps), whereas I want the aircraft to fly ultimately for 10m (6000 time steps). I am thinking that it is too difficult to train a network to fly for such a long period of time because it would have to learn all the states leading up to that point.

Results
I found that training the algorithm typically resulted in extreme volatility in the test simulation's score (sum of all rewards at every time step). The score history would look like: (moving average of 100 scores in orange).

What I've tried

1. Changing the standard deviation of the normal distribution of the actor output so that there is less exploration (stability of the aircraft is quite sensitive to the particular control values).
   
2. Advantage normalization
3. Increasing training time to 10 000 simulations (~40 000 learning iterations since there are 4 epochs per simulation)
4. Testing the algorithm on the gym's cartpole environment (discrete actions). I was able to successfully train the actor in 200ish games.

If anyone has any other suggestions of things to try, it would be greatly appreciated.

Answer 1

Training using only 20 timesteps at a time is far too small, especially when the goal will ultimately consist of episodes of length 6000. You definitely need to increase that substantially and that will probably solve your problem immediately. You might try something like simulate 5 episodes and then train on all timesteps in those 5 episodes.

If that still doesn't work, another thing you can do is try training the value function (critic) differently, e.g. use monte carlo returns or more steps in the temporal difference, and you can keep a memory replay for the value function as well.

Answer 2

A different, variable reward structure might help. You could try a combination of airspeed, pitch, roll and whether it is hovering in the air or not in each timestep as a representation for the reward.

Maybe airspeed should, in expectation, contribute up to 30% of the reward, pitch up to 15%, roll up to 15% and being in the air up to 40%. This would explicitly reward motion, trying new movements as well as being reasonable, i.e. hovering in the air. You can create a new formula for the reward around this premise, even use some logarithms or other fancy functions that you see fit.

The important thing is that this way, the reward in each timestep is varying based on the four aforementioned variables and the chosen formula. It is not 0 or 1 like your previous structures. There is also feedback in each timestep on what it is doing right and how well it is doing, not just the last timestep like the second structure.