I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.

  • $\begingroup$ Do you understand how does epsilon-greedy help exploration in Q-learning for example? Because the idea is basically the same. $\endgroup$
    – Brale
    Commented Feb 23, 2020 at 15:23
  • $\begingroup$ Yes, in Q-learning you have discrete actions and we choose actions randomly during exploration and then we choose the action that gives the highest value. Let me explain my confusion by this example. Lets say my actor network generated a force of 1 N, but I need 10 N for the same state so I added big noise to make the action almost 10 N. Now the 10 N and the state go into the critic network and it will give a good value and the actor will know that 1 N is enough, but it will not know that I need 10 not 1. $\endgroup$ Commented Feb 24, 2020 at 7:13
  • $\begingroup$ That's not true because you're not updating actor through action that was modified by noise, you're updating it through action that it wanted to initially take. So in your case the update would be through action 1N and not 10N. Only critic is updated through noise modified action. The reason why this helps exploration is because the critic will eventually get more accurate in representing state-action values so it will "criticize" the actor better through updates $\endgroup$
    – Brale
    Commented Feb 24, 2020 at 16:43
  • $\begingroup$ Got it ! I will try it out and revert back to you. Thank you $\endgroup$ Commented Feb 25, 2020 at 13:57
  • $\begingroup$ Hi, I hope you are doing great. I tried to update the critic through the noisy actions and the actor through the action its self. However, the actor is not learning, the only thing that changed is that the actions are shifted like all actions become shifted by 2N then 5N and so on. Please not that I am applying the noise for some episodes. $\endgroup$ Commented Mar 29, 2020 at 11:59


You must log in to answer this question.

Browse other questions tagged .