Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
From what I understand, if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided towards its learning goal.
Are the above thoughts correct, and are there any other pros and cons of the two contrasting settings? On a side-note, I feel that the inability to specify rewards that are dense in time is what makes imitation learning useful.
What are the pros and cons of sparse and dense rewards in reinforcement learning?
It is unusual to refer to this difference as "pros and cons" because that term is often used to make comparisons between difference choices. Assuming you have a specific problem to solve, then whether or not the rewards are naturally sparse or dense is not a choice. You cannot say "I want to solve MountainCar, I will use a dense reward setting", because MountainCar has (relatively, for a starting problem) sparse rewards. You can only say "I won't attempt MountainCar, it is too difficult".
In short however, your assessment is correct:
if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided towards its learning goal
There is not really any other difference at the top level. Essentially, sparser rewards make for a harder problem to solve. All RL algorithms can cope with sparse rewards to some degree, the whole concept of returns and value backup is designed to deal with sparseness at a theoretical level. In practical terms however, it may take some algorithms an unreasonable amount of time to determine a good policy beyond certain levels of sparseness.
On a side-note, I feel that the inability to specify rewards that are dense in time is what makes imitation learning useful.
Imitation learning is one of many techniques available to work around or deal with problems that have sparse reward structure. Others include:
Reward shaping, which attempts to convert a sparse reward scheme to a dense one using domain knowledge of the researcher.
Eligibility traces, which back up individual TD errors across multiple time steps.
Prioritised sweeping, which focuses updates on "surprising" reward data.
Action selection planning algorithms that look ahead from the current state.
"Curiousity" driven reinforcement learning that guides exploration to new state spaces independently of any reward signal.
