3
$\begingroup$

I am trying to use Deep-Q-Learning to learn an ANN which controls a 7-DOF robotic arm. The robotic arm must avoid an obstacle and reach a target.

I have implemented a number of state-of-art techinques to try to improve the ANN performance. Such techniques are: PER, Double DQN, adaptive discount factor, sparse reward. I have also tried Dueling DQN but it performed poorly. I have also tried a number of ANN architectures and it looks like that 2 hidden layers with 128 neurons is the best one so far. My input layer is 12 neurons, the output 10 neurons.

However, as you can see from the image down here, at a certain point the DQN stops learning and gets stuck at around 80% of success rate. I don't understand why it gets stuck, because in my opinion we could reach an higher success rate, 90% at least, but I just can't get out of that "local minimum".

so, my question is: What are some techniques I can try to unstuck a DQN from something that looks like a local minimum?

figure:

enter image description here

note: the success rate in this picture is computed as the number of successes in the last 100 runs.

$\endgroup$
  • $\begingroup$ Is your success rate measured during training runs, or a separate assessment using greedy actions only? What is your exploration technique - if $\epsilon$-greedy, what is $\epsilon$? $\endgroup$ – Neil Slater Apr 22 at 10:32
  • 1
    $\begingroup$ Hi @Neil Slater. It is measured during training. My exploration rule is epsilon greedy with espilon starting at 1 and decaying to 0.01 with a decay factor of 0.9995 $\endgroup$ – olinarr Apr 22 at 12:36
  • $\begingroup$ Thanks for that detail. It is not possible to say what variation caused by a 0.01 exploration rate would make to your environment. Could you please test one later (mid-80% ) success rate agents, using $\epsilon = 0$, with, say 1000 runs, and report the success rate? If it is close to 80%, then the environment and/or agent are robust to mistakes. If it is notably higher, then this is a better/true measure of the performance of your agent after training $\endgroup$ – Neil Slater Apr 22 at 17:41
  • $\begingroup$ yes already tested that with 10k runs: 78% success rate with eps = 0. I'm sorry but I didn't understand the rest of the comment: could you explain again? $\endgroup$ – olinarr Apr 22 at 17:51
  • 1
    $\begingroup$ I meant that the measurement is necessary, because even a small exploration rate can adversely affect success during training. This is very dependent on the nature of the environment. A "robust" (my term, not standard) environment here is one where one or two mistakes don't matter much and can be corrected. Which in turn means that your graph during training is more usable to help rack possible fixes $\endgroup$ – Neil Slater Apr 22 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.