I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.

  • $\begingroup$ Do you understand how does epsilon-greedy help exploration in Q-learning for example? Because the idea is basically the same. $\endgroup$ – Brale Feb 23 '20 at 15:23
  • $\begingroup$ Yes, in Q-learning you have discrete actions and we choose actions randomly during exploration and then we choose the action that gives the highest value. Let me explain my confusion by this example. Lets say my actor network generated a force of 1 N, but I need 10 N for the same state so I added big noise to make the action almost 10 N. Now the 10 N and the state go into the critic network and it will give a good value and the actor will know that 1 N is enough, but it will not know that I need 10 not 1. $\endgroup$ – Ahmad Fares Feb 24 '20 at 7:13
  • $\begingroup$ That's not true because you're not updating actor through action that was modified by noise, you're updating it through action that it wanted to initially take. So in your case the update would be through action 1N and not 10N. Only critic is updated through noise modified action. The reason why this helps exploration is because the critic will eventually get more accurate in representing state-action values so it will "criticize" the actor better through updates $\endgroup$ – Brale Feb 24 '20 at 16:43
  • $\begingroup$ Got it ! I will try it out and revert back to you. Thank you $\endgroup$ – Ahmad Fares Feb 25 '20 at 13:57
  • $\begingroup$ Hi, I hope you are doing great. I tried to update the critic through the noisy actions and the actor through the action its self. However, the actor is not learning, the only thing that changed is that the actions are shifted like all actions become shifted by 2N then 5N and so on. Please not that I am applying the noise for some episodes. $\endgroup$ – Ahmad Fares Mar 29 '20 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.