Skip to main content

Questions tagged [sparse-rewards]

For questions about the sparsity of the rewards (or reward function), which can slow down learning. Reward shaping can be used to solve this problem.

Filter by
Sorted by
Tagged with
0 votes
1 answer
31 views

A question about the reward calculation in the Hindsight Experience Replay algorithm

I' m try to implement the HER algorithm from scratch in order to use it in the PandaReach-v3 environment. I already developed the same algorithm for the bitflip environment and it works as expected. ...
Dave's user avatar
  • 182
2 votes
0 answers
51 views

How does PPO with advantage normalization learn in MountainCar-v0 before first reaching the goal state?

I'm trying to figure out how PPO ever learns anything in a sparse environment like gymnasium's MountainCar-v0 before it first ever reaches the goal state. Specifically was looking at stable_baselines3'...
Switch's user avatar
  • 121
0 votes
1 answer
77 views

How do I update Q-values in Q-learning when rewards may only be received after many actions?

I am working on a Q-learning system where the agent may well (and almost always) have to take many actions before a reward can be given to the agent (or more so, the notion of a reward in my context ...
Hera Sutton's user avatar
0 votes
0 answers
42 views

Multi-Agent DQN not learning for Clean Up Game - Reward slowly decreasing

The environment of the Clean Up game is simple: in a 25*18 grid world, there's dirt spawning on the left side and apples spawning on the other. Agents get a +1 reward for eating an apple (by stepping ...
Charles's user avatar
0 votes
0 answers
504 views

How does Proximal Policy Optimization deal with sparse reward

In the original paper, the objective of PPO is as follows:. My question is, how does this objective behave in a sparse reward setting (i.e., reward is only given after a sequence of actions were taken)...
Sam's user avatar
  • 195
1 vote
1 answer
538 views

How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?

Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is ...
cerebrou's user avatar
  • 151
6 votes
1 answer
1k views

How to improve the reward signal when the rewards are sparse?

In cases where the reward is delayed, this can negatively impact a models ability to do proper credit assignment. In the case of a sparse reward, are there ways in which this can be negated? In a ...
tryingtolearn's user avatar
4 votes
2 answers
1k views

How to apply Q-learning when rewards is only available at the last state?

I have a scheduling problem in which there are $n$ slots and $m$ clients. I am trying to solve the problem using Q-learning so I have made the following state-action model. A state $s_t$ is given by ...
zdm's user avatar
  • 301
6 votes
1 answer
3k views

What are the pros and cons of sparse and dense rewards in reinforcement learning?

From what I understand, if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided ...
stoic-santiago's user avatar
5 votes
1 answer
244 views

How does the optimization process in hindsight experience replay exactly work?

I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the ...
vikram71198's user avatar
3 votes
1 answer
494 views

Are there any reliable ways of modifying the reward function to make the rewards less sparse?

If I am training an agent to try and navigate a maze as fast as possible, a simple reward would be something like \begin{align} R(\text{terminal}) &= N - \text{time}\ \ , \ \ N \gg \text{...
Paradox's user avatar
  • 133
4 votes
1 answer
3k views

Can reinforcement learning be used for tasks where only one final reward is received?

Is reinforcement learning problem adaptable to the setting when there is only one - final - reward. I am aware of problems with sparse and delayed rewards, but what about only one reward and a quite ...
TomR's user avatar
  • 863