Skip to main content

Questions tagged [hindsight-experience-replay]

For questions about the Hindsight Experience Replay (HER), proposed in the paper "Hindsight Experience Replay" (2017) by Marcin Andrychowicz et al.

Filter by
Sorted by
Tagged with
0 votes
0 answers
20 views

A question about the goal selection in the second loop in the HER algorithm

I have to admit, that I still have problem digesting the HER algorithm proposed in the famous paper and despite the fact, that the idea behind it should be intuitive, I realize that it is not so easy ...
Dave's user avatar
  • 194
0 votes
1 answer
34 views

A question about the reward calculation in the Hindsight Experience Replay algorithm

I' m try to implement the HER algorithm from scratch in order to use it in the PandaReach-v3 environment. I already developed the same algorithm for the bitflip environment and it works as expected. ...
Dave's user avatar
  • 194
1 vote
2 answers
64 views

Why does HER not work with on-policy RL algorithms?

I'm wondering because I don't appreciate what is wrong with just applying HER to an otherwise on-policy algorithm? Like if we do that will the training stability just fall apart? And if so why? My ...
profPlum's user avatar
  • 434
0 votes
0 answers
20 views

How to establish baselines, with different training loops

My objective is to test out a new algorithm that I designed. However, I am confused whether my methodology to train the networks is correct. I am just concerned about the training loops: In the first ...
Yash_Bit's user avatar
3 votes
0 answers
445 views

How can PPO be combined with HER?

I ask because PPO is apparently an on-policy algorithm & the HER paper says that it can be combine with any off-policy algorithm. Yet I see GitHub projects that have combined them somehow? How is ...
profPlum's user avatar
  • 434
1 vote
1 answer
114 views

What does $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ mean in the article Hindsight Experience Replay, section 2.1?

Taken from section 2.1 in the article: We consider the standard reinforcement learning formalism consisting of an agent interacting with an environment. To simplify the exposition we assume that the ...
WinnieThePooh's user avatar
1 vote
0 answers
214 views

Why would DDPG with Hindsight Experience Replay not converge?

I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
Vedant Shah's user avatar
1 vote
1 answer
96 views

What do the state features of KukaGymEnv represent? [closed]

I trying to use DDPG augmented with Hindsight Experience Replay (HER) on pybullet's KukaGymEnv. To formulate the feature vector for the goal state, I need to know what the features of the state of the ...
Vedant Shah's user avatar
3 votes
0 answers
235 views

Is this a good approach to solving Atari's "Montezuma's Revenge"?

I'm new to Reinforcement Learning. For an internship, I am currently training Atari's "Montezuma's Revenge" using a double Deep Q-Network with Hindsight Experience Replay (HER) (see also ...
vikram71198's user avatar
5 votes
1 answer
245 views

How does the optimization process in hindsight experience replay exactly work?

I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the ...
vikram71198's user avatar
2 votes
1 answer
276 views

What is the difference between success rate and reward when dealing with binary and sparse rewards?

In OpenAI Gym "reward" is defined as: reward (float): amount of reward achieved by the previous action. The scale varies between environments, but the ...
rrz0's user avatar
  • 263
2 votes
0 answers
62 views

How does Hindsight Experience Replay cope with multiple goals?

What if there are multiple goals? For example, let's consider the bit-flipping environment as described in the paper HER with one small change: now, the goal is not some specific configuration, but ...
Savco's user avatar
  • 61
3 votes
1 answer
279 views

How does Hindsight Experience Replay learn from unsuccessful trajectories?

I am confused by how HER learns from unsuccessful trajectories. I understand that from failed trajectories it creates 'fake' goals that it can learn from. Ignoring HER for now, if in the case where ...
piccolo's user avatar
  • 173