Skip to main content

Questions tagged [reward-functions]

For questions about rewards functions (e.g. in the context of reinforcement learning, which may be denoted as $R(s, a)$).

Filter by
Sorted by
Tagged with
13 votes
3 answers
2k views

Why is the reward in reinforcement learning always a scalar?

I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
Sid Mani's user avatar
  • 233
12 votes
4 answers
2k views

Counterexamples to the reward hypothesis

On Sutton and Barto's RL book, the reward hypothesis is stated as that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative ...
Bananin's user avatar
  • 221
8 votes
2 answers
11k views

What are some best practices when trying to design a reward function?

Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward ...
12 rhombi in grid w no corners's user avatar
8 votes
1 answer
7k views

Suitable reward function for trading buy and sell orders

I am working to build a deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are ...
fgauth's user avatar
  • 189
8 votes
2 answers
3k views

How do we define the reward function for an environment?

How do you actually decide what reward value to give for each action in a given state for an environment? Is this purely experimental and down to the programmer of the environment? So, is it a ...
Hazzaldo's user avatar
  • 289
8 votes
1 answer
3k views

What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?

I created an OpenAI Gym environment, and I would like to check the performance of the agent from OpenAI Baselines DQN approach on it. In my environment, the best possible outcome for the agent is 0 - ...
AlexGuevara's user avatar
7 votes
2 answers
2k views

How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?

In this video, the lecturer states that $R(s)$, $R(s, a)$ and $R(s, a, s')$ are equivalent representations of the reward function. Intuitively, this is the case, according to the same lecturer, ...
nbro's user avatar
  • 40.9k
7 votes
1 answer
4k views

Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?

If we shift the rewards by any constant (which is a type of reward shaping), the optimal state-action value function (and so optimal policy) does not change. The proof of this fact can be found here. ...
nbro's user avatar
  • 40.9k
6 votes
1 answer
1k views

How to improve the reward signal when the rewards are sparse?

In cases where the reward is delayed, this can negatively impact a models ability to do proper credit assignment. In the case of a sparse reward, are there ways in which this can be negated? In a ...
tryingtolearn's user avatar
6 votes
1 answer
3k views

What are the pros and cons of sparse and dense rewards in reinforcement learning?

From what I understand, if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided ...
stoic-santiago's user avatar
6 votes
2 answers
2k views

Why does shifting all the rewards have a different impact on the performance of the agent?

I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
Fishfish's user avatar
6 votes
1 answer
808 views

How should I handle invalid actions in a grid world?

I'm building a really simple experiment, where I let an agent move from the bottom-left corner to the upper-right corner of a $3 \times 3$ grid world. I plan to use DQN to do this. I'm having trouble ...
o_yeah's user avatar
  • 197
5 votes
2 answers
1k views

Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?

Sutton and Barto define the state–action–next-state reward function, $r(s, a, s')$, as follows (equation 3.6, p. 49) $$ r(s, a, s^{\prime}) \doteq \mathbb{E}\left[R_{t} \mid S_{t-1}=s, A_{t-1}=a, S_{t}...
SAGALPREET SINGH's user avatar
5 votes
1 answer
1k views

How do I convert an MDP with the reward function in the form $R(s,a,s')$ to and an MDP with a reward function in the form $R(s,a)$?

The AIMA book has an exercise about showing that an MDP with rewards of the form $r(s, a, s')$ can be converted to an MDP with rewards $r(s, a)$, and to an MDP with rewards $r(s)$ with equivalent ...
Asher's user avatar
  • 436
5 votes
1 answer
244 views

How define a reward function for a humanoid agent whose goal is to stand up from the ground?

I'm trying to teach a humanoid agent how to stand up after falling. The episode starts with the agent lying on the floor with its back touching the ground, and its goal is to stand up in the shortest ...
Tirafesi's user avatar
  • 151
4 votes
2 answers
117 views

Can rewards be decomposed into components?

I'm training a robot to walk to a specific $(x, y)$ point using TD3, and, for simplicity, I have something like ...
pinkie pAI's user avatar
4 votes
1 answer
2k views

How can I ensure convergence of DDQN, if the true Q-values for different actions in the same state are very close?

I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future ...
apitsch's user avatar
  • 93
4 votes
2 answers
1k views

How to apply Q-learning when rewards is only available at the last state?

I have a scheduling problem in which there are $n$ slots and $m$ clients. I am trying to solve the problem using Q-learning so I have made the following state-action model. A state $s_t$ is given by ...
zdm's user avatar
  • 301
4 votes
1 answer
718 views

Can the rewards be stochastic when the transition model is deterministic?

Suppose we have a deterministic environment where knowing $s,a$ determines $s'$. Is it possible to get two different rewards $r\neq r'$ in some state $s_{\text{fixed}}$? Assume that $s_{\text{fixed}}$ ...
rewardbabe's user avatar
4 votes
1 answer
2k views

Is a reward given at every step or only given when the RL agent fails or succeeds?

In reinforcement learning, an agent can receive a positive reward for correct actions and a negative reward for wrong actions, but does the agent also receive rewards for every other step/action?
Dan D.'s user avatar
  • 1,293
4 votes
0 answers
75 views

What research has been done on learning non-Markovian reward functions?

Recently, some work has been done planning and learning in Non-Markovian Decision Processes, that is, decision-making with temporally extended rewards. In these settings, a particular reward is ...
Gavin Rens's user avatar
3 votes
1 answer
2k views

How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?

Is there any empirical/theoretical evidence on the effect of initial values of state-action and state values on the training of an RL agent (the values an RL agent assigns to visited states) via MC ...
user avatar
3 votes
1 answer
691 views

Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

I have implemented a simple Q-learning algorithm to minimize a cost function by setting the reward to the inverse of the cost of the action taken by the agent. The algorithm converges nicely, but ...
EArwa's user avatar
  • 77
3 votes
1 answer
163 views

Reward design or Inverse reinforcement learning?

I'm working on a reinforcement learning project where I only have demonstrations (i.e. set of states and actions). During my research on how handle the reward signal, I noticed that research papers ...
Eman.suradi's user avatar
3 votes
1 answer
356 views

Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward

In the appendix of the Constrained Policy Optimization (CPO) paper (Arxiv), the authors denote the discounted future state distribution $d^\pi$ as: $$d^\pi(s) = (1-\gamma) \sum_{t=0}^\infty{\gamma^t P(...
josealeixo.pc's user avatar
3 votes
3 answers
1k views

Is the policy really invariant under affine transformations of the reward function?

In the context of a Markov decision process, this paper says it is well-known that the optimal policy is invariant to positive affine transformation of the reward function On the other hand, ...
IssaRice's user avatar
  • 171
3 votes
1 answer
398 views

How can I implement the reward function for an 8-DOF robot arm with TRPO?

I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to ...
user1690356's user avatar
3 votes
0 answers
154 views

Is better to reward short- or long-term progress in Q-learning?

I have been training some kind of agent to reach a target using a Q-learning based approach, and I have tried two different types of rewards: Long-term reward: $\mathrm{reward} = - \mathrm{distance}(\...
Thomas Wagenaar's user avatar
2 votes
1 answer
192 views

Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true?

I am referring to eq. 3.6 (page 49) based on Sutton's online book and can be found in an image below. I could not make sense of the final derivation of the equation $r(s, a, s')$. My question is ...
alfa_80's user avatar
  • 123
2 votes
1 answer
998 views

How to deal with small reward values

In my environment rewards are generally small, e.g. [-0.01, 0.01]. My concern is that small reward values might get dominated or distorted by the noise during the training. Does it make sense to scale ...
Mika's user avatar
  • 341
2 votes
2 answers
87 views

If $p(s'|s,a) = 0$, would the reward the reward $r(s,a,s')$ be infinite? [duplicate]

In chapter 2 of Barto and Sutton's RL book, the four argument probability function $p: S \times R \times S \times A \to [0,1]$ is reduced to three arguments $p: S \times S \times A \to [0,1]$ as ...
Jahid Chowdhury Choton's user avatar
2 votes
1 answer
483 views

In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?

In the context of my problem, the "true" reward is not additive. Realistically, the more reward the agent has already accumulated, the easier it becomes to accumulate even more. That's to ...
Vladimir Belik's user avatar
2 votes
1 answer
233 views

Can the rewards be matrices when using DQN?

I have a basic question. I'm working towards developing a reward function for my DQN. I'd like to train an RL agent to edit pixels on an image. I understand that convolutions are ideal for working ...
junfanbl's user avatar
  • 323
2 votes
1 answer
148 views

How to construct a reward function for a "wait and see" problem

I'm working on a problem that I think could probably be represented as a reinforcement learning task, but I'm uncertain about how to design the reward function. The core task is essentially a ...
user336650's user avatar
2 votes
1 answer
145 views

Is there a mathematical formalism to deal with a missing reward signal?

Typically, a Reinforcement Learning learning problem is formalized as finding an optimal policy for a Markov Decision Process (MDP). In many real-life situations, however, an agent can only get ...
Onil90's user avatar
  • 183
2 votes
1 answer
757 views

What happens with policy gradient methods if rewards are differentiable?

I would like some help with understanding why there is no explicit flow of information from the reward gradient to the parameters of the policy in policy gradient methods. What I mean is the following,...
External Supplier Staff's user avatar
2 votes
2 answers
564 views

How should I define the reward function to solve the Wumpus game with deep Q-learning?

I'm writing a DQN agent for the Wumpus game. Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...
Edwin Carlsson's user avatar
2 votes
1 answer
598 views

Why does potential-based reward shaping seem to alter the optimal policy in this case?

It is known that every potential function won't alter the optimal policy [1]. I lack of understanding why is that. The definition: $$R' = R + F,$$ with $$F = \gamma\Phi(s') - \Phi(s),$$ where, let's ...
ScientiaEtVeritas's user avatar
2 votes
1 answer
177 views

Where are the parentheses in the definition of $r(s,a)$?

I am new to RL and I am trying to work through the book Reinforcement Learning: An Introduction I (Sutton & Barto, 2018). In chapter 3 on Finite Markov Decision Processes, the authors write the ...
MrYouMath's user avatar
  • 255
2 votes
1 answer
578 views

What are proxy reward functions?

The understanding I have is that they somehow adjust the objective to make it easier to meet, without changing the reward function. ... the observed proxy reward function is the approximate solution ...
mugoh's user avatar
  • 531
2 votes
1 answer
271 views

What is the difference between success rate and reward when dealing with binary and sparse rewards?

In OpenAI Gym "reward" is defined as: reward (float): amount of reward achieved by the previous action. The scale varies between environments, but the ...
rrz0's user avatar
  • 263
2 votes
2 answers
341 views

How to define a reward function in POMDPs?

How do I define a reward function for my POMDP model? In the literature, it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this ...
Bryan McGill's user avatar
2 votes
0 answers
74 views

In RL, is the quantification of the reward function arbitrary? Does it affect the learning?

There are different ways to set the reward function, such as extrinsic (externally provided rewards), intrinsic (the rewards are generated by the agents themselves based on their internal state and ...
John Prada's user avatar
2 votes
0 answers
271 views

How to solve a reinforcement learning problem with changing rewards?

I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for ...
Zhenzhen Gong's user avatar
2 votes
1 answer
1k views

How to encourage the reinforcement-learning agent to reach the goal as quickly as possible, and what's the effect of discount factor?

I am trying to use reinforcement learning to solve a task and compare its performance to humans. The task is to find a single target in a fixed number of locations. At each step, the agent will pick ...
Cloudy's user avatar
  • 223
2 votes
1 answer
150 views

How can I go from $R(s)$ to $R(s,a)$ in this specific MDP?

I'm trying to implement a research paper, as explained in this other post, here the author of the paper assumed R as a function of both states and actions, while the code (and the MDP) I'm using to ...
ИванКарамазов's user avatar
2 votes
0 answers
34 views

How can I discourage the RL agent from drawing in a zero-sum game?

My agent receives $1, 0, -1$ rewards for winning, drawing, and losing the game, respectively. What would be the consequences of setting reward to $-1$ for draws? Would that encourage the agent to win ...
mark mark's user avatar
  • 773
2 votes
0 answers
119 views

How to combine two differently equally important signals into the reward function, that have different scales?

I have two signals that I want to use to model my reward. The first one is the CPU TIME: running mean from this diagram: The second one is the MAX RESIDUAL from this diagram: Since they are both ...
tmaric's user avatar
  • 392
1 vote
1 answer
282 views

Is it necessary to have a constant reward in the terminal state?

I have downloaded the grid world project form this link. I have executed the project multiple times using: python gridworld.py -k 20 -a q -r -0.2 -s 90 I have ...
AAA's user avatar
  • 111
1 vote
1 answer
225 views

Is there a reward function that would encourage exploration in this case?

I am new to Reinforcement Learning. I am trying to train PPO agent for citylearn. The goal is to lower two environmental variables from observations. The default reward function is ...
Sai Dinesh Pola's user avatar