1
$\begingroup$

I am using KerasRL DDPG to try to learn a policy on my own custom environment, but the agent is stuck in a local optima although I am adding the OrnsteinUhlenbeck randomization process. I used the exact same DDPG to solve Pendulum-v0 and it works, but my environment is a more complex with a continuous space/action space.

How do you deal with local optima problem in reinforcement learning? is it just an exploitation issue?

More details:

My state space is not pixels, it is numerical, in fact it's a metro line simulator and the state space is the velocity, the position of each train on the line and the number of passengers at each station. I need to control the different trains so I am not trying to control only one train but all the operational trains and each one can have different actions such as speed or not, stay longer on the next station or not etc.

1/ I am using the same ANN architecture for the actor and critic: 3 FC layers with (512, 256, 256) hidden units.

2/ Adam optimizer for both the actor and critic with a small lr=1e-7 and clipnorm=1.

3/ nb_steps_warmup_critic=1000, nb_steps_warmup_actor=1000

4/ SequentialMemory(limit=1000000, window_length=1)

5/ The environement is a simulator of a metro line with a continuous state and action space

$\endgroup$
11
  • $\begingroup$ Hi @BAKYAC. Given that there's no logical bug in the algorithm/program, it could come down to fine-tuning the hyperparameters (learning rate, ann architecture, etc.). As I understand, finding a perfect combination of those can be quite challenging. There are some vague general guidelines, but still, the hyperparameters are usually specific for each particular environment. If you elaborate in your question on what kind of architecture you have, and what kind of problem you're trying to solve, you might get some input from other users (though this SE is somewhat in dormant state). $\endgroup$ – mark mark Dec 18 '20 at 15:10
  • $\begingroup$ Thank you for your suggestions! I edited my question and yes I do think it's a hyperparameters fine-tuning problem but as the training takes too long I am not able to test out different parameters ! $\endgroup$ – BAKYAC Dec 19 '20 at 13:44
  • $\begingroup$ You could try using Google Colab – they let you use their GPU and TPU for free, though there's a limit. That really helped me to speed up model training and tuning $\endgroup$ – mark mark Dec 19 '20 at 20:20
  • $\begingroup$ I have an NVIDIA Quadro T1000 GPU so I don't think I need google colab as they provide a tesla k80, plus the training crash after some time and mine can take days to finish.. $\endgroup$ – BAKYAC Dec 20 '20 at 10:13
  • 1
    $\begingroup$ Oh no I am not working with pixels. I edited my question.. thanks anyway ! $\endgroup$ – BAKYAC Dec 20 '20 at 21:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.