5 votes
Accepted

What are some best practices when trying to design a reward function?

Designing reward functions Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three ...
user avatar
  • 33k
4 votes

What are some best practices when trying to design a reward function?

If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with ...
user avatar
  • 337
4 votes
Accepted

How do we define the reward function for an environment?

In Reinforcement Learning (RL), a reward function is part of the problem definition and should: Be based primarily on the goals of the agent. Take into account any combination of starting state $s$, ...
user avatar
  • 23.1k
4 votes
Accepted

Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?

I believe the claim is true. Here is my attempt at a proof. Let us consider the optimal infinite horizon value function $V_\alpha^*$ of $\mathcal{M}_\alpha$ at an arbitrary state $s \in S$. The value $...
user avatar
  • 371
3 votes
Accepted

How to deal with changing environment in reinforcement learning

I am correct in my understanding that you only provide the agent with the state of the car, i.e. a global x and y position, its angle, velocity, and steering angle? How does the agent know that it is ...
user avatar
  • 159
3 votes
Accepted

What should I do when the potential value of a state is too high?

Dennis Soemers provides an important point that from a theoretical standpoint, this can be seen as a non-issue. However, what you bring up is an important practical issue of potential-based reward ...
user avatar
3 votes

What should I do when the potential value of a state is too high?

I don't think the situation you're sketching should be a problem at all. If $P(s)$ is high (e.g. $P(s) = 1000$), this means (according to your shaping / "heuristic") that it's valuable to be in the ...
user avatar
  • 9,316
3 votes
Accepted

Are there any reliable ways of modifying the reward function to make the rewards less sparse?

Doing something like the dense, distance-based reward signal you propose is possible... but you have to do it very carefully. If you're not careful, and do it in a naive manner, you are likely to ...
user avatar
  • 9,316
2 votes
Accepted

Why does potential-based reward shaping seem to alter the optimal policy in this case?

The same $\gamma = 0.9$ that you use in the definition $F \doteq \gamma \Phi(s') - \Phi(s)$ should also be used as the discount factor in computing returns for multi-step trajectories. So, rather than ...
user avatar
  • 9,316
2 votes

Is the policy really invariant under affine transformations of the reward function?

This statement: (it is well-known that the optimal policy is invariant to positive affine transformation of the reward function). is, as far as I know, and as you summarise, incorrect* because ...
user avatar
  • 23.1k
2 votes

How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?

Sutton and Barto state, "The reward signal is your way of communicating to the robot [agent] what you want it to achieve, not how you want it achieved." Since you stated that the goal is to reach the ...
user avatar
  • 776
2 votes
Accepted

Is a reward given at every step or only given when the RL agent fails or succeeds?

In reinforcement learning (RL), an immediate reward value must be returned after each action, along with the next state. This value can be zero though, which will have no direct impact on optimality ...
user avatar
  • 23.1k
1 vote

How to deal with changing environment in reinforcement learning

My guess is that you haven't trained long enough, but there are things that can be done to possibly accelerate learning. It depends on what you want the policy to do in the final version. If you want ...
user avatar
  • 46
1 vote

How would you shape a reward function if there was four quantities to optimize?

Here is how I managed to construct a reward function in one of my projects, where I trained an RL model for a self-driving robot that has only a single camera to navigate through a tunnel: $$ R = \...
user avatar
1 vote

Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?

Your examples are equivalent. But it is possible to find a constant yielding a different optimal policy. Your examples are absolutely equivalent. The agent maximizes the reward, and only way to do so ...
user avatar
1 vote
Accepted

How can I fix jerky movement in a continuous action space

I think you should try to reason in terms of total "area" explored by the agent rather than "how far" it moves from the initial point, and also you should add some reward terms to ...
user avatar
1 vote

Why does shifting all the rewards have a different impact on the performance of the agent?

You have some freedom to re-define reward schemes, whilst still describing the same goals for an agent. How this works depends to some degree on whether you are dealing with an episodic or continuing ...
user avatar
  • 23.1k
1 vote

Is the policy really invariant under affine transformations of the reward function?

In framing the problem as an episodic reinforcement learning problem, the goal is to find a policy that optimizes $\mathbb{E}[\sum_{t=0}^\tau r(s_t)],$ where $\tau$ is the random time at which the ...
user avatar
1 vote

Can recovering a reward function using IRL lead to better policies compared to reward shaping?

Inverse Reinforcement Learning (IRL) is a technique that attempts to recover the reward function that the expert is implicitly maximising based on expert demonstrations. When solving reinforcement ...
user avatar
  • 1,201
1 vote

How do we define the reward function for an environment?

Yes, you are right. It is somehow an arbitrary choice, although you should consider the reasonable numerical ranges of your activation functions if you decide to go beyond the values +/- 1. You can ...
user avatar
  • 583
1 vote
Accepted

Expressing Arbitrary Reward Functions as Potential-Based Advice (PBA)

Is the method itself defective or anything wrong with my code? There does indeed appear to be an issue with the code, the publications are fine (I know most of those authors and would very much trust ...
user avatar
  • 9,316

Only top scored, non community-wiki answers of a minimum length are eligible