Questions tagged [reward-shaping]
For questions related to reward shaping, which is a technique where supplemental rewards are provided to make a problem easier to learn. In general, there is usually an obvious natural reward for any problem. For games, this is usually a win or loss. For financial problems, the reward is usually profit. Reward shaping augments the natural reward signal by adding additional rewards for making progress toward a good solution.
24
questions
2
votes
2
answers
217
views
One to one relation between state + action -> reward
I am designing my own environment for a specific problem and I am thinking of the reward function for it. In some RL algos it is common to learn the reward that is associated with taking an action ...
- 61
1
vote
0
answers
226
views
MDP with a non-markovian reward function?
I have set up a RL environment and it converges to a decent solution when using a reward function:
$R(s_t,a_t) = fenv(s_t, a_t)$ , where $fenv$ is the environment dynamics.
Now, i want to change the ...
- 11
0
votes
0
answers
34
views
Reward Function for Reinforcement Learning model
I am trying to create a reinforcement learning model to control the acceleration of a car. I am designing the model such that initially the acceleration is provided and then deceleration is provided ...
0
votes
0
answers
124
views
Shaping reward so that it maximizes multiple components together
I am fairly new to RL and I compete in AWS DeepRacer student league. The main task there is to create a reward function. All the hyperparameters and action space are fixed. So far, I know how to shape ...
2
votes
1
answer
739
views
How to deal with small reward values
In my environment rewards are generally small, e.g. [-0.01, 0.01]. My concern is that small reward values might get dominated or distorted by the noise during the training. Does it make sense to scale ...
- 341
2
votes
0
answers
252
views
How to solve a reinforcement learning problem with changing rewards?
I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for ...
3
votes
2
answers
2k
views
How to deal with changing environment in reinforcement learning
I am new to RL and I'm currently working on implementing a DQN and DDPG agent for a 2D car parking environment. I want to train my agent so that it can successfully traverse the env and park in the ...
0
votes
1
answer
267
views
How would you shape a reward function if there was four quantities to optimize?
I found this article quite useful on how to shape a reward function in RL. However, the example they gave is quite simple, where the goal is to minimize only two quantities (velocity and distance).
...
- 36
6
votes
1
answer
147
views
Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?
Say I've got two Markov Decision Processes (MDPs):
$$\mathcal{M_0} = (\mathcal{S}, \mathcal{A}, P, R_0),\quad\text{and}\quad\mathcal{M}_1 = (\mathcal{S}, \mathcal{A}, P, R_1)$$
Both have the same set ...
- 2,426
7
votes
1
answer
3k
views
Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?
If we shift the rewards by any constant (which is a type of reward shaping), the optimal state-action value function (and so optimal policy) does not change. The proof of this fact can be found here.
...
- 39.6k
3
votes
1
answer
250
views
How can I fix jerky movement in a continuous action space
I am training an agent to do object avoidance. The agent has control over its steering angle and its speed. The steering angle and speed are normalized in a $[−1,1]$ range, where the sign encodes ...
- 65
7
votes
2
answers
8k
views
What are some best practices when trying to design a reward function?
Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward ...
4
votes
1
answer
2k
views
Is a reward given at every step or only given when the RL agent fails or succeeds?
In reinforcement learning, an agent can receive a positive reward for correct actions and a negative reward for wrong actions, but does the agent also receive rewards for every other step/action?
- 1,283
6
votes
2
answers
2k
views
Why does shifting all the rewards have a different impact on the performance of the agent?
I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
- 61
3
votes
1
answer
907
views
How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?
I'm building an agent for a racing game. In this game, there is a randomized map where there are speed boosts for the player to pick up and obstacles that act to slow the player down. The goal of the ...
- 31
3
votes
3
answers
931
views
Is the policy really invariant under affine transformations of the reward function?
In the context of a Markov decision process, this paper says
it is well-known that the optimal policy is invariant to positive affine transformation of the reward function
On the other hand, ...
- 171
4
votes
1
answer
227
views
Can recovering a reward function using IRL lead to better policies compared to reward shaping?
I am working on a research project about the different reward functions being used in the RL domain. I have read up on Inverse Reinforcement Learning (IRL) and Reward Shaping (RS). I would like to ...
- 1,251
4
votes
1
answer
440
views
How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?
I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (in $[-1, 1]$).
To this end, I use a ...
- 171
3
votes
1
answer
417
views
Are there any reliable ways of modifying the reward function to make the rewards less sparse?
If I am training an agent to try and navigate a maze as fast as possible, a simple reward would be something like
\begin{align}
R(\text{terminal}) &= N - \text{time}\ \ , \ \ N \gg \text{...
- 133
5
votes
1
answer
185
views
How define a reward function for a humanoid agent whose goal is to stand up from the ground?
I'm trying to teach a humanoid agent how to stand up after falling. The episode starts with the agent lying on the floor with its back touching the ground, and its goal is to stand up in the shortest ...
- 151
8
votes
2
answers
2k
views
How do we define the reward function for an environment?
How do you actually decide what reward value to give for each action in a given state for an environment?
Is this purely experimental and down to the programmer of the environment? So, is it a ...
- 279
3
votes
1
answer
154
views
Expressing Arbitrary Reward Functions as Potential-Based Advice (PBA)
I am trying to reproduce the results for the simple grid-world environment in [1]. But it turns out that using a dynamically learned PBA makes the performance worse and I cannot obtain the results ...
- 33
2
votes
1
answer
511
views
Why does potential-based reward shaping seem to alter the optimal policy in this case?
It is known that every potential function won't alter the optimal policy [1]. I lack of understanding why is that.
The definition:
$$R' = R + F,$$ with $$F = \gamma\Phi(s') - \Phi(s),$$
where, let's ...
3
votes
2
answers
2k
views
What should I do when the potential value of a state is too high?
I'm working on a Reinforcement Learning task where I use reward shaping as proposed in the paper Policy invariance under reward transformations:
Theory and application to reward shaping (1999) by ...
- 185