Questions tagged [reward-design]
For questions about designing (or defining) reward functions e.g. for reinforcement learning problems.
53
questions
0
votes
0
answers
34
views
How can I improve these reinforcement learning reward functions?
I am working for the very first time to a RL problem. More in details, I am working with a 3DOF model of a buisness jet aircraft. The environement is made up by the aircraft model + the two PID ...
- 1
1
vote
1
answer
53
views
Why should the agent bounce the ball back and forth on the same side of the screen in Atari Breakout?
The following is from page 17 of "Michael Hu, “The Art of Reinforcement Learning: Fundamentals, Mathematics, and Implementations with Python”, Apress, 2023"
https://link.springer.com/book/...
- 973
0
votes
1
answer
65
views
Which RL algorithms can be used in an environment where actions have to be performed only in specific situations?
I am wondering which RL algorithms can be used in an environment where actions have to be performed only in specific situations. For example, on a conveyor belt on which a box that fulfills certain ...
- 223
1
vote
1
answer
70
views
What is the name of the reward function that utilizes the rewards of the next n steps?
I have a problem with continuous time, observation and action space. I am discretizing the time to be able to apply the usual Reinforcement Learning algorithms (I chose PPO). The problem consists of a ...
0
votes
0
answers
69
views
How to handle penalty and reward occurring simultaneously
Assume the following scenario:
We have an agent that acts on an environment where the agent should never take an action that results in him leaving the environment. For example, imagine an agent ...
- 195
3
votes
1
answer
107
views
Why doesn't Reward Normalization subtract the mean?
I want to introduce Reward Normalization for my current RL project. For this, I looked at how other libraries implemented this functionality. I found the following implementation:
https://github.com/...
- 195
2
votes
0
answers
75
views
In RL, is the quantification of the reward function arbitrary? Does it affect the learning?
There are different ways to set the reward function, such as extrinsic (externally provided rewards), intrinsic (the rewards are generated by the agents themselves based on their internal state and ...
- 21
1
vote
1
answer
244
views
Is there a reward function that would encourage exploration in this case?
I am new to Reinforcement Learning. I am trying to train PPO agent for citylearn. The goal is to lower two environmental variables from observations. The default reward function is
...
0
votes
0
answers
35
views
Reward Function for Reinforcement Learning model
I am trying to create a reinforcement learning model to control the acceleration of a car. I am designing the model such that initially the acceleration is provided and then deceleration is provided ...
3
votes
1
answer
169
views
Reward design or Inverse reinforcement learning?
I'm working on a reinforcement learning project where I only have demonstrations (i.e. set of states and actions). During my research on how handle the reward signal, I noticed that research papers ...
- 133
0
votes
2
answers
240
views
Is there a way to form a reward function so that it would take into account the order of the actions?
I want to design a multi-arm bandit system for a multi-step, multi-location system. Locations are dynamic, so I can not design the system based on them. In each location, the alternative actions that ...
1
vote
0
answers
38
views
How to teach DRL agent to avoid erratic, high frequency behavior (when costs for taking actions are already included)?
I have a situation where an agent can take actions to enter a state where rewards can be obtained (or costs can be incurred), but the actions themselves have a small cost. In my reward function, I ...
- 362
0
votes
0
answers
264
views
How to solve a reinforcement learning problem with a stochastic reward function?
In a discrete time system, an environment has an unknown reward probability $p(r|s,a)$. However, the transition probability $p(s'\mid s,a)$ is deterministic.
In my case, the reward for the same action ...
2
votes
1
answer
526
views
In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?
In the context of my problem, the "true" reward is not additive. Realistically, the more reward the agent has already accumulated, the easier it becomes to accumulate even more. That's to ...
- 362
2
votes
1
answer
164
views
How to construct a reward function for a "wait and see" problem
I'm working on a problem that I think could probably be represented as a reinforcement learning task, but I'm uncertain about how to design the reward function. The core task is essentially a ...
- 21
1
vote
1
answer
270
views
How should I write the reward function to teach the agent the rules of this card game?
I'm quite new to reinforcement learning. I've been training the model for the following problem but the mean reward is stuck.
In a 5 by 5 board, each position can contain a card with a color (0-4) ...
- 113
0
votes
1
answer
73
views
Deep RL reward design for neuron centerline extraction task
As part of a bigger scope project, I'm training a RL agent that attempts to reconstruct, pixel by pixel, the trajectory of a neuron on a segmented image. To give a better insight on the task that I'm ...
- 1
1
vote
0
answers
81
views
How to teach Machine Learning Agent to destroy replicating objects in a puzzle game?
I have an unusual but very interesting problem. I have a game that is very similar to Toon Blast (a puzzle mobile game). It's based on a Match-2 mechanic in which you can destroy 2 or more connected ...
- 153
1
vote
1
answer
2k
views
How should I define the reward function for a stock trading-like game?
Problem setting
Consider a game like trading a stock
At each step, the agent can buy / sell a stock.
Trade is a pair of ...
- 155
0
votes
1
answer
312
views
How would you shape a reward function if there was four quantities to optimize?
I found this article quite useful on how to shape a reward function in RL. However, the example they gave is quite simple, where the goal is to minimize only two quantities (velocity and distance).
...
- 36
0
votes
0
answers
60
views
How to choose the reward in reinforcement learning? [duplicate]
I am solving a combinatorial optimization problem, where I do not have a global optimum, so the goal is to improve the objective function as much as possible. So, to do this, I was inspired by this ...
- 41
2
votes
1
answer
257
views
Can the rewards be matrices when using DQN?
I have a basic question. I'm working towards developing a reward function for my DQN. I'd like to train an RL agent to edit pixels on an image. I understand that convolutions are ideal for working ...
- 323
3
votes
2
answers
2k
views
Reinforcement Learning algorithm with rewards dependent both on previous action and current action
Problem description:
Suppose we have an environment, where a reward at time step $t$ is dependent not only on the current action, but also on previous action in the following way:
if current action ==...
- 33
3
votes
0
answers
162
views
Is better to reward short- or long-term progress in Q-learning?
I have been training some kind of agent to reach a target using a Q-learning based approach, and I have tried two different types of rewards:
Long-term reward: $\mathrm{reward} = - \mathrm{distance}(\...
- 1,217
1
vote
1
answer
258
views
If the reward function of an environment depends on some initial conditions, should I create a separate environment for each condition?
I would like some guidance on how to design an Environment for a Reinforcement Learning agent where the stopping conditions and rewards for the environment change based on an initial set of input ...
- 15
0
votes
0
answers
68
views
Thompson sampling with Bernoulli prior and non-binary reward update
I am solving a problem for which I have to select the best possible servers (level 1) to hit for a given data. These servers (level 1) in turn hit some other servers (level 2) to complete the request. ...
- 101
2
votes
0
answers
35
views
How can I discourage the RL agent from drawing in a zero-sum game?
My agent receives $1, 0, -1$ rewards for winning, drawing, and losing the game, respectively. What would be the consequences of setting reward to $-1$ for draws? Would that encourage the agent to win ...
- 793
2
votes
2
answers
577
views
How should I define the reward function to solve the Wumpus game with deep Q-learning?
I'm writing a DQN agent for the Wumpus game.
Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...
3
votes
1
answer
537
views
How do I design the rewards and penalties for an agent whose goal it is to explore a map
I am trying to train an agent to explore an unknown two-dimensional map while avoiding circular obstacles (with varying radii). The agent has control over its steering angle and its speed. The ...
- 65
2
votes
0
answers
124
views
How to combine two differently equally important signals into the reward function, that have different scales?
I have two signals that I want to use to model my reward.
The first one is the CPU TIME: running mean from this diagram:
The second one is the MAX RESIDUAL from this diagram:
Since they are both ...
- 392
8
votes
2
answers
12k
views
What are some best practices when trying to design a reward function?
Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward ...
- 195
4
votes
1
answer
2k
views
Is a reward given at every step or only given when the RL agent fails or succeeds?
In reinforcement learning, an agent can receive a positive reward for correct actions and a negative reward for wrong actions, but does the agent also receive rewards for every other step/action?
- 1,293
4
votes
2
answers
117
views
Can rewards be decomposed into components?
I'm training a robot to walk to a specific $(x, y)$ point using TD3, and, for simplicity, I have something like ...
- 45
1
vote
0
answers
300
views
Designing a reward function for my reinforcement learning problem
I'm working on a project lately and I'm trying to solve a problem with reinforcement learning and I have serious issues with shaping the reward function.
The problem is designing a device with maximum ...
- 11
6
votes
1
answer
847
views
How should I handle invalid actions in a grid world?
I'm building a really simple experiment, where I let an agent move from the bottom-left corner to the upper-right corner of a $3 \times 3$ grid world.
I plan to use DQN to do this. I'm having trouble ...
- 197
3
votes
1
answer
709
views
Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?
I have implemented a simple Q-learning algorithm to minimize a cost function by setting the reward to the inverse of the cost of the action taken by the agent. The algorithm converges nicely, but ...
- 77
3
votes
1
answer
1k
views
How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?
I'm building an agent for a racing game. In this game, there is a randomized map where there are speed boosts for the player to pick up and obstacles that act to slow the player down. The goal of the ...
- 31
3
votes
1
answer
429
views
How can I implement the reward function for an 8-DOF robot arm with TRPO?
I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to ...
- 31
4
votes
1
answer
578
views
How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?
I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (in $[-1, 1]$).
To this end, I use a ...
- 171
12
votes
4
answers
2k
views
Counterexamples to the reward hypothesis
On Sutton and Barto's RL book, the reward hypothesis is stated as
that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative ...
- 221
3
votes
1
answer
535
views
Are there any reliable ways of modifying the reward function to make the rewards less sparse?
If I am training an agent to try and navigate a maze as fast as possible, a simple reward would be something like
\begin{align}
R(\text{terminal}) &= N - \text{time}\ \ , \ \ N \gg \text{...
- 133
3
votes
1
answer
2k
views
How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?
Is there any empirical/theoretical evidence on the effect of initial values of state-action and state values on the training of an RL agent (the values an RL agent assigns to visited states) via MC ...
user9947
5
votes
1
answer
278
views
How define a reward function for a humanoid agent whose goal is to stand up from the ground?
I'm trying to teach a humanoid agent how to stand up after falling. The episode starts with the agent lying on the floor with its back touching the ground, and its goal is to stand up in the shortest ...
- 151
8
votes
2
answers
3k
views
How do we define the reward function for an environment?
How do you actually decide what reward value to give for each action in a given state for an environment?
Is this purely experimental and down to the programmer of the environment? So, is it a ...
- 289
4
votes
1
answer
3k
views
Can reinforcement learning be used for tasks where only one final reward is received?
Is reinforcement learning problem adaptable to the setting when there is only one - final - reward. I am aware of problems with sparse and delayed rewards, but what about only one reward and a quite ...
- 863
4
votes
1
answer
167
views
Expressing Arbitrary Reward Functions as Potential-Based Advice (PBA)
I am trying to reproduce the results for the simple grid-world environment in [1]. But it turns out that using a dynamically learned PBA makes the performance worse and I cannot obtain the results ...
- 43
5
votes
1
answer
725
views
Can the rewards be stochastic when the transition model is deterministic?
Suppose we have a deterministic environment where knowing $s,a$ determines $s'$. Is it possible to get two different rewards $r\neq r'$ in some state $s_{\text{fixed}}$? Assume that $s_{\text{fixed}}$ ...
- 51
8
votes
1
answer
7k
views
Suitable reward function for trading buy and sell orders
I am working to build a deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are ...
- 189
2
votes
1
answer
635
views
How should I define the reward function in the case of Connect Four?
I'm using RL to train a Network on the game Connect4. It learns quickly that 4 connected pieces is good. It gets a reward of 1 for this. A zero is rewarded for all other moves.
It takes quite a time ...
- 123
2
votes
2
answers
357
views
How to define a reward function in POMDPs?
How do I define a reward function for my POMDP model?
In the literature, it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this ...
- 441