I am trying to understand some of the different approaches used to overcome sparse rewards in a reinforcement learning setting for a research project. Particularly, I have looked at curiosity driven learning, where an agent learns an intrinsic reward function based on the uncertainty of the next state that the agent will end up in as he takes action a in state s. The greater the uncertainty of the next state, the higher the rewards. This will incentive agent's to be more exploratory and it is used particularly in some games where a huge number of steps is needed before the agent reaches the terminal state where is he only then rewarded.

The curiosity driven approach as demonstrated in this paper: https://pathak22.github.io/noreward-rl/ is able to learn faster than if a 0 rewards were used for each state, action.

To my knowledge, using different reward functions will affect the optimal policy obtained. Would curiosity driven learning therefore lead to a different policy as compared to whether a 0 reward was used ? Assume that for a 0 immediate reward system, it is able to derive a policy that reaches the goal state. Which of these 2 policies will be more optimal ?

  • $\begingroup$ The goal of curiosity driven learning is obviously to get better average original reward for trained policy. Global maximum on complete policy space for original reward will be bigger or same, but (if curiosity driven learning really works) trained approximation from curiosity could be better then trained approximation from original reward. $\endgroup$ – mirror2image Feb 12 '20 at 6:33
  • $\begingroup$ @mirror2image That is practically an answer, consider making it one $\endgroup$ – Neil Slater Feb 12 '20 at 8:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.