3
$\begingroup$

From what I understand, if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided towards its learning goal.

Are the above thoughts correct, and are there any other pros and cons of the two contrasting settings? On a side-note, I feel that the inability to specify rewards that are dense in time is what makes imitation learning useful.

$\endgroup$
2
$\begingroup$

What are the pros and cons of sparse and dense rewards in reinforcement learning?

It is unusual to refer to this difference as "pros and cons" because that term is often used to make comparisons between difference choices. Assuming you have a specific problem to solve, then whether or not the rewards are naturally sparse or dense is not a choice. You cannot say "I want to solve MountainCar, I will use a dense reward setting", because MountainCar has (relatively, for a starting problem) sparse rewards. You can only say "I won't attempt MountainCar, it is too difficult".

In short however, your assessment is correct:

if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided towards its learning goal

There is not really any other difference at the top level. Essentially, sparser rewards make for a harder problem to solve. All RL algorithms can cope with sparse rewards to some degree, the whole concept of returns and value backup is designed to deal with sparseness at a theoretical level. In practical terms however, it may take some algorithms an unreasonable amount of time to determine a good policy beyond certain levels of sparseness.

On a side-note, I feel that the inability to specify rewards that are dense in time is what makes imitation learning useful.

Imitation learning is one of many techniques available to work around or deal with problems that have sparse reward structure. Others include:

  • Reward shaping, which attempts to convert a sparse reward scheme to a dense one using domain knowledge of the researcher.

  • Eligibility traces, which back up individual TD errors across multiple time steps.

  • Prioritised sweeping, which focuses updates on "surprising" reward data.

  • Action selection planning algorithms that look ahead from the current state.

  • "Curiousity" driven reinforcement learning that guides exploration to new state spaces independently of any reward signal.

$\endgroup$
  • 1
    $\begingroup$ Could you point me in the direction of interesting papers related to the techniques you've mentioned, besides imitation learning? That'd be helpful! $\endgroup$ – strawberry-sunshine Aug 13 '20 at 7:49
  • 1
    $\begingroup$ @cogito_ai: Not really, most are broad topics and you are better off using a web search to find tutorials. For eligibility traces and planning, Sutton & Barto has a good chapter on each. It also has a section on prioritised sweeping in planning context (DQN may also use it to select replay memory). Here's something on cuirosity: towardsdatascience.com/… - there were a few papers on that in 2018 $\endgroup$ – Neil Slater Aug 13 '20 at 7:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.