Questions tagged [reward-shaping]

For questions related to reward shaping, which is a technique where supplemental rewards are provided to make a problem easier to learn. In general, there is usually an obvious natural reward for any problem. For games, this is usually a win or loss. For financial problems, the reward is usually profit. Reward shaping augments the natural reward signal by adding additional rewards for making progress toward a good solution.

Filter by
Sorted by
Tagged with
6
votes
1answer
166 views

Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?

If we shift the rewards by any constant (which is a type of reward shaping), the optimal state-action value function (and so optimal policy) does not change. The proof of this fact can be found here. ...
0
votes
1answer
111 views

What is the optimal value function of the shifted version of the reward function?

Similarly to this question that I asked some time ago, what is the optimal value function of the shifted (by some constant $c$) version of some reward function? More precisely, let's assume that $r(s, ...
3
votes
1answer
82 views

How can I fix jerky movement in a continuous action space

I am training an agent to do object avoidance. The agent has control over its steering angle and its speed. The steering angle and speed are normalized in a $[−1,1]$ range, where the sign encodes ...
4
votes
2answers
268 views

What are some best practices when trying to design a reward function?

Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward ...
2
votes
1answer
218 views

Is a reward given at every step or only given when the RL agent fails or succeeds?

In reinforcement learning, an agent can receive a positive reward for correct actions and a negative reward for wrong actions, but does the agent also receive rewards for every other step/action?
4
votes
1answer
194 views

Why does shifting all the rewards have a different impact on the performance of the agent?

I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
3
votes
1answer
116 views

How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?

I'm building an agent for a racing game. In this game, there is a randomized map where there are speed boosts for the player to pick up and obstacles that act to slow the player down. The goal of the ...
3
votes
1answer
101 views

Can recovering a reward function using IRL lead to better policies compared to reward shaping?

I am working on a research project about the different reward functions being used in the RL domain. I have read up on Inverse Reinforcement Learning (IRL) and Reward Shaping (RS). I would like to ...
3
votes
1answer
126 views

How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?

I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (in $[-1, 1]$). To this end, I use a ...
2
votes
1answer
279 views

What is the optimal value function of the scaled version of the reward function?

Consider the reward function $r(s, a)$ with optimal state-action value function $q_*(s, a)$. What would be the optimal state-action value function of $c r(s, a)$, for $c \in \mathbb{R}$? Would it be $...
3
votes
1answer
64 views

Are there any reliable ways of modifying the reward function to make the rewards less sparse?

If I am training an agent to try and navigate a maze as fast as possible, a simple reward would be something like \begin{align} R(\text{terminal}) &= N - \text{time}\ \ , \ \ N \gg \text{...
4
votes
0answers
52 views

How define a reward function for a humanoid agent whose goal is to stand up from the ground?

I'm trying to teach a humanoid agent how to stand up after falling. The episode starts with the agent lying on the floor with its back touching the ground, and its goal is to stand up in the shortest ...
6
votes
2answers
542 views

How do we define the reward function for an environment?

How do you actually decide what reward value to give for each action in a given state for an environment? Is this purely experimental and down to the programmer of the environment? So, is it a ...
3
votes
1answer
73 views

Expressing Arbitrary Reward Functions as Potential-Based Advice (PBA)

I am trying to reproduce the results for the simple grid-world environment in [1]. But it turns out that using a dynamically learned PBA makes the performance worse and I cannot obtain the results ...
2
votes
1answer
53 views

Why does potential-based reward shaping seem to alter the optimal policy in this case?

It is known that every potential function won't alter the optimal policy [1]. I lack of understanding why is that. The definition: $$R' = R + F,$$ with $$F = \gamma\Phi(s') - \Phi(s),$$ where, let's ...
3
votes
2answers
1k views

What should I do when the potential value of a state is too high?

I'm working on a Reinforcement Learning task where I use reward shaping as proposed in the paper Policy invariance under reward transformations: Theory and application to reward shaping (1999) by ...