0
$\begingroup$

I' m try to implement the HER algorithm from scratch in order to use it in the PandaReach-v3 environment. I already developed the same algorithm for the bitflip environment and it works as expected. So, what's now the problem? The problem is the calculation of the new reward $r'$ given an additional goal as stated in the red box in the following picture:

enter image description here

The problem is, that while in the bitflip environment I just created a trivial function, that basically calculates the difference between the state $s_{t}$ and the goal $g'$, in the case of a more complex problem like the PandaReach I don't kwon how to proceed, since I cannot figure out, how the function for the calculation of the new reward given a new goal should look like.

Basically I came up with the following two ideas:

  1. I could implement a function, which calculates the 3D distance between the state and the goal. The function could output a one, if the calculated distance is below a small $\epsilon$ or zero otherwise;

  2. I could use the already implemented step method (from the environment), which delivers in one step the new calculated reward. So no effort from my side. But here the problem is that I need to pass an action to the step method in order to generate a new state and calculate the new reward. And, for a new action, I need a policy. Since HER is a off policy algorithm I have a behavioural policy $\pi_{b}$ and nothing else. So my concern is whether I can/should/may use the same behavioural policy to sample an action to feed into the step method and get the new reward.

In other words: how do you calculate the new reward $r'$ in the HER algorithm?

Many thanks

Improve this question
$\endgroup$
1
  • $\begingroup$ HER typically assumes you have access to a reward function that can evaluate whether a certain state achieves the goal or not. If you can implement 1. then that should work. $\endgroup$
    – David
    Commented Mar 19 at 11:25

1 Answer 1

Reset to default
1
$\begingroup$

Meanwhile I can answer my own question. After figuring it out, what the HER algorithm really needs, I found out, that the right answer is 1: you need a reward function for calculating the new reward and the new done flag.

In case you are at the first tries and you are trying to implement it using the panda-gym environment, this is what I did:

Inside the Panda environment there is a compute_reward function, which can be used to calculate the reward. Just pass the actual state and the new goal to that function.

In code:

import gymnasium as gyms import panda_gym

... 
env = gyms.make('PandaReach-v3')

...

# The compute_reward functions need an 'info' dictionary as third argument.
# Just pass an empty dictionary there.
# 'next_state' should replace with the variable you need. Since in my case
# I want to calculate the difference between 3 spatial coordinates, so I'm going to take 'next_state' and 'new_goal'. Both are 3D spatial coordinates 
new_reward = env.unwrapped.compute_reward(next_state, new_goal, {})
# Or in case you are not using any wrapper new_reward = 
env.computer_reward(next_state, new_goal, {})
Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .