I am using clipped PPO to train a neural network to act as the controller for steering an aircraft, and am finding that my networks aren't learning. The goal is to keep the aircraft flying to cover the distance, and the code is implemented using Pytorch. I am wondering if anyone who is more experienced would be willing to take a look at my implementation.
Aircraft and actor model
I have a flight simulation environment with the dynamical system of an aircraft, where the simulation is propagated through time in 0.1s increments. The states of the aircraft serve as inputs to the actor network (airspeed, pitch, heading, height), and the controls include pitch and roll angles.
The actor network uses a multivariate normal distribution to output two values that serve as the means for the pitch and roll angle controls. The variances are fixed. This way, the policy is stochastic and allows for exploration. I am using two hidden layers of 256 neurons each for both actor and critic networks, with learning rates of 1e-4. ReLu activation on the hidden layers and sigmoid on the actor's output, with the critic's output being linear.
Reward structure
I have tried two different reward structures:
- A reward of 1 for every time step that the aircraft is in the air (with a reward of 0 in the terminal state, which is when the aircraft crashes to the ground or if it achieves the maximum flight time)
- A reward of 0 for every time step, but a single reward computed at the terminal state that is proportional to the displacement of the aircraft over the simulation, and an extremely large penalty that is subtracted from this reward if the aircraft crashes.
Neither of these reward structures seemed to allow learning. I am wondering if simply having a reward of 1 at every time step, much like the gym cartpole environment is insufficient in getting the model to learn how to fly. I was expecting the network to figure out ways of controlling the aircraft to fly longer to obtain a greater total reward. For the second reward structure, I expected the advantage estimation to carry the large reward/penalty backward through the trajectory so that the actor would learn to avoid flying in certain ways that end up in crashing the plane.
PPO implementation
For the clipped PPO implementation, I am:
- Resetting the flight simulation environment
- Propagating the simulation until the aircraft crashes or reaches maximum flight time
- Training the actor and critic networks every 20 time steps (2s of simulation)
- Repeating this for a maximum number of simulations
For training, I take the 20 time steps, shuffle them, and divide them into 5 batches. Then I train the network on these batches, reshuffle the 20 time steps and create 5 new batches, and train again. I repeat this process for a total of 4 epochs per learning cycle.
The fastest that the aircraft can reach the ground (crash) is 1.7s of simulation time (17 time steps), whereas I want the aircraft to fly ultimately for 10m (6000 time steps). I am thinking that it is too difficult to train a network to fly for such a long period of time because it would have to learn all the states leading up to that point.
Results
I found that training the algorithm typically resulted in extreme volatility in the test simulation's score (sum of all rewards at every time step). The score history would look like: (moving average of 100 scores in orange).
What I've tried
- Changing the standard deviation of the normal distribution of the actor output so that there is less exploration (stability of the aircraft is quite sensitive to the particular control values).
- Advantage normalization
- Increasing training time to 10 000 simulations (~40 000 learning iterations since there are 4 epochs per simulation)
- Testing the algorithm on the gym's cartpole environment (discrete actions). I was able to successfully train the actor in 200ish games.
If anyone has any other suggestions of things to try, it would be greatly appreciated.