9 votes
Accepted

What are some best practices when trying to design a reward function?

Designing reward functions Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three ...
nbro's user avatar
  • 40.6k
6 votes

What are some best practices when trying to design a reward function?

If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with ...
user76284's user avatar
  • 347
5 votes
Accepted

How to deal with changing environment in reinforcement learning

I am correct in my understanding that you only provide the agent with the state of the car, i.e. a global x and y position, its angle, velocity, and steering angle? How does the agent know that it is ...
Lars's user avatar
  • 189
5 votes
Accepted

How do we define the reward function for an environment?

In Reinforcement Learning (RL), a reward function is part of the problem definition and should: Be based primarily on the goals of the agent. Take into account any combination of starting state $s$, ...
Neil Slater's user avatar
  • 32.1k
4 votes
Accepted

Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?

I believe the claim is true. Here is my attempt at a proof. Let us consider the optimal infinite horizon value function $V_\alpha^*$ of $\mathcal{M}_\alpha$ at an arbitrary state $s \in S$. The value $...
mikkola's user avatar
  • 579
4 votes
Accepted

How to deal with small reward values

The numbers that a value-based neural network will predict are usually based on expected returns (sum of rewards by end of an episode, or a discounted infinite sum), although in some cases they might ...
Neil Slater's user avatar
  • 32.1k
3 votes

One to one relation between state + action -> reward

Maybe not clear from the other answer, but no, the reward depends usually also on the state you end up, which is not deterministic by definition of MDPs, and even in that case, the reward can be noisy....
Alberto's user avatar
  • 1,915
3 votes

One to one relation between state + action -> reward

The RL framework assumes only that the reward function depends on the current state, the selected action, and the next state: $$\mathcal{R}(s_t, a_t, s_{t+1})$$ It can be deterministic but it can also ...
pi-tau's user avatar
  • 807
3 votes
Accepted

Are there any reliable ways of modifying the reward function to make the rewards less sparse?

Doing something like the dense, distance-based reward signal you propose is possible... but you have to do it very carefully. If you're not careful, and do it in a naive manner, you are likely to ...
Dennis Soemers's user avatar
  • 10.3k
3 votes
Accepted

Is a reward given at every step or only given when the RL agent fails or succeeds?

In reinforcement learning (RL), an immediate reward value must be returned after each action, along with the next state. This value can be zero though, which will have no direct impact on optimality ...
Neil Slater's user avatar
  • 32.1k
3 votes
Accepted

What should I do when the potential value of a state is too high?

Dennis Soemers provides an important point that from a theoretical standpoint, this can be seen as a non-issue. However, what you bring up is an important practical issue of potential-based reward ...
Brenden Petersen's user avatar
3 votes

What should I do when the potential value of a state is too high?

I don't think the situation you're sketching should be a problem at all. If $P(s)$ is high (e.g. $P(s) = 1000$), this means (according to your shaping / "heuristic") that it's valuable to be in the ...
Dennis Soemers's user avatar
  • 10.3k
2 votes
Accepted

Expressing Arbitrary Reward Functions as Potential-Based Advice (PBA)

Is the method itself defective or anything wrong with my code? There does indeed appear to be an issue with the code, the publications are fine (I know most of those authors and would very much trust ...
Dennis Soemers's user avatar
  • 10.3k
2 votes
Accepted

Why does potential-based reward shaping seem to alter the optimal policy in this case?

The same $\gamma = 0.9$ that you use in the definition $F \doteq \gamma \Phi(s') - \Phi(s)$ should also be used as the discount factor in computing returns for multi-step trajectories. So, rather than ...
Dennis Soemers's user avatar
  • 10.3k
2 votes

Is the policy really invariant under affine transformations of the reward function?

This statement: (it is well-known that the optimal policy is invariant to positive affine transformation of the reward function). is, as far as I know, and as you summarise, incorrect* because ...
Neil Slater's user avatar
  • 32.1k
2 votes

Can recovering a reward function using IRL lead to better policies compared to reward shaping?

Inverse Reinforcement Learning (IRL) is a technique that attempts to recover the reward function that the expert is implicitly maximising based on expert demonstrations. When solving reinforcement ...
calveeen's user avatar
  • 1,261
2 votes

Why does shifting all the rewards have a different impact on the performance of the agent?

You have some freedom to re-define reward schemes, whilst still describing the same goals for an agent. How this works depends to some degree on whether you are dealing with an episodic or continuing ...
Neil Slater's user avatar
  • 32.1k
2 votes

How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?

Sutton and Barto state, "The reward signal is your way of communicating to the robot [agent] what you want it to achieve, not how you want it achieved." Since you stated that the goal is to reach the ...
DeepQZero's user avatar
  • 1,379
1 vote
Accepted

What is the name of the reward function that utilizes the rewards of the next n steps?

Your function could be called the truncated return - i.e. the sum of rewards up to some time step in the future. It would be unusual to perform reward shaping by taking the orginal reward from an ...
Neil Slater's user avatar
  • 32.1k
1 vote

How to deal with changing environment in reinforcement learning

My guess is that you haven't trained long enough, but there are things that can be done to possibly accelerate learning. It depends on what you want the policy to do in the final version. If you want ...
Elfurd's user avatar
  • 46
1 vote

How would you shape a reward function if there was four quantities to optimize?

Here is how I managed to construct a reward function in one of my projects, where I trained an RL model for a self-driving robot that has only a single camera to navigate through a tunnel: $$ R = \...
Aray Karjauv's user avatar
1 vote

Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?

Your examples are equivalent. But it is possible to find a constant yielding a different optimal policy. Your examples are absolutely equivalent. The agent maximizes the reward, and only way to do so ...
BlueMoon93's user avatar
1 vote
Accepted

How can I fix jerky movement in a continuous action space

I think you should try to reason in terms of total "area" explored by the agent rather than "how far" it moves from the initial point, and also you should add some reward terms to ...
Edoardo Guerriero's user avatar
1 vote

Why does shifting all the rewards have a different impact on the performance of the agent?

Our paper 'Exploit Reward Shifting in Value-Based DRL' answered this question. In the case mentioned, using a negative shift will lead to explorative behaviors, therefore the DQN agent has better ...
user68435's user avatar
1 vote

Is the policy really invariant under affine transformations of the reward function?

In framing the problem as an episodic reinforcement learning problem, the goal is to find a policy that optimizes $\mathbb{E}[\sum_{t=0}^\tau r(s_t)],$ where $\tau$ is the random time at which the ...
Dylan Hadfield-Menell's user avatar
1 vote
Accepted

How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?

After some research on the subject, I found a possible solution to my problem of high frequency oscillations in continuous control using DDPG: I added a reward component based on the actuator ...
opt12's user avatar
  • 171
1 vote

How do we define the reward function for an environment?

Yes, you are right. It is somehow an arbitrary choice, although you should consider the reasonable numerical ranges of your activation functions if you decide to go beyond the values +/- 1. You can ...
DrMcCleod's user avatar
  • 603

Only top scored, non community-wiki answers of a minimum length are eligible