I am using KerasRL DDPG to try to learn a policy on my own custom environment, but the agent is stuck in a local optima although I am adding the OrnsteinUhlenbeck randomization process. I used the exact same DDPG to solve Pendulum-v0 and it works, but my environment is a more complex with a continuous space/action space.
How do you deal with local optima problem in reinforcement learning? is it just an exploitation issue?
My state space is not pixels, it is numerical, in fact it's a metro line simulator and the state space is the velocity, the position of each train on the line and the number of passengers at each station. I need to control the different trains so I am not trying to control only one train but all the operational trains and each one can have different actions such as speed or not, stay longer on the next station or not etc.
1/ I am using the same ANN architecture for the actor and critic: 3 FC layers with (512, 256, 256) hidden units.
2/ Adam optimizer for both the actor and critic with a small lr=1e-7 and clipnorm=1.
3/ nb_steps_warmup_critic=1000, nb_steps_warmup_actor=1000
4/ SequentialMemory(limit=1000000, window_length=1)
5/ The environement is a simulator of a metro line with a continuous state and action space