Questions tagged [reward-functions]
For questions about rewards functions (e.g. in the context of reinforcement learning, which may be denoted as $R(s, a)$).
39
questions
2
votes
1answer
54 views
Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward
In the appendix of the Constrained Policy Optimization (CPO) paper (Arxiv), the authors denote the discounted future state distribution $d^\pi$ as:
$$d^\pi(s) = (1-\gamma) \sum_{t=0}^\infty{\gamma^t P(...
5
votes
1answer
103 views
How to improve the reward signal when the rewards are sparse?
In cases where the reward is delayed, this can negatively impact a models ability to do proper credit assignment. In the case of a sparse reward, are there ways in which this can be negated?
In a ...
2
votes
1answer
51 views
Can the rewards be matrices when using DQN?
I have a basic question. I'm working towards developing a reward function for my DQN. I'd like to train an RL agent to edit pixels on an image. I understand that convolutions are ideal for working ...
2
votes
1answer
63 views
How can I go from $R(s)$ to $R(s,a)$ in this specific MDP?
I'm trying to implement a research paper, as explained in this other post, here the author of the paper assumed R as a function of both states and actions, while the code (and the MDP) I'm using to ...
1
vote
1answer
143 views
What is the difference between a fitness function and a reward function?
In reinforcement learning (RL), the reward function (RF), which can be denoted as $r(s)$, $r(s, a)$, $r(s, a, s')$, $r(s, s')$ depending on its specific definition, provides the learning signal, which ...
2
votes
0answers
26 views
Is better to reward short- or long-term progress in Q-learning?
I have been training some kind of agent to reach a target using a Q-learning based approach, and I have tried two different types of rewards:
Long-term reward: $\mathrm{reward} = - \mathrm{distance}(\...
0
votes
0answers
52 views
Is my reward function non-Markovian?
I am working on an RL problem where the time when the agent obtains the reward for taking action $a$ in time step $t$ is stochastic. In fact, there is no immediate reward for taking action $a$ in time ...
1
vote
1answer
51 views
If the reward function of an environment depends on some initial conditions, should I create a separate environment for each condition?
I would like some guidance on how to design an Environment for a Reinforcement Learning agent where the stopping conditions and rewards for the environment change based on an initial set of input ...
0
votes
0answers
36 views
Thompson sampling with Bernoulli prior and non-binary reward update
I am solving a problem for which I have to select the best possible servers (level 1) to hit for a given data. These servers (level 1) in turn hit some other servers (level 2) to complete the request. ...
2
votes
0answers
22 views
How can I discourage the RL agent from drawing in a zero-sum game?
My agent receives $1, 0, -1$ rewards for winning, drawing, and losing the game, respectively. What would be the consequences of setting reward to $-1$ for draws? Would that encourage the agent to win ...
6
votes
1answer
171 views
Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?
If we shift the rewards by any constant (which is a type of reward shaping), the optimal state-action value function (and so optimal policy) does not change. The proof of this fact can be found here.
...
0
votes
1answer
112 views
What is the optimal value function of the shifted version of the reward function?
Similarly to this question that I asked some time ago, what is the optimal value function of the shifted (by some constant $c$) version of some reward function? More precisely, let's assume that $r(s, ...
2
votes
2answers
144 views
How should I define the reward function to solve the Wumpus game with deep Q-learning?
I'm writing a DQN agent for the Wumpus game.
Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...
2
votes
1answer
101 views
What are proxy reward functions?
The understanding I have is that they somehow adjust the objective to make it easier to meet, without changing the reward function.
... the observed proxy reward function is the approximate solution ...
3
votes
2answers
106 views
How to apply Q-learning when rewards is only available at the last state?
I have a scheduling problem in which there are $n$ slots and $m$ clients. I am trying to solve the problem using Q-learning so I have made the following state-action model.
A state $s_t$ is given by ...
3
votes
1answer
430 views
What are the pros and cons of sparse and dense rewards in reinforcement learning?
From what I understand, if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided ...
2
votes
0answers
35 views
How to combine two differently equally important signals into the reward function, that have different scales?
I have two signals that I want to use to model my reward.
The first one is the CPU TIME: running mean from this diagram:
The second one is the MAX RESIDUAL from this diagram:
Since they are both ...
9
votes
3answers
358 views
Why is the reward in reinforcement learning always a scalar?
I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
4
votes
2answers
280 views
What are some best practices when trying to design a reward function?
Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward ...
2
votes
1answer
232 views
Is a reward given at every step or only given when the RL agent fails or succeeds?
In reinforcement learning, an agent can receive a positive reward for correct actions and a negative reward for wrong actions, but does the agent also receive rewards for every other step/action?
3
votes
2answers
89 views
Can rewards be decomposed into components?
I'm training a robot to walk to a specific $(x, y)$ point using TD3, and, for simplicity, I have something like ...
4
votes
1answer
209 views
Why does shifting all the rewards have a different impact on the performance of the agent?
I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
5
votes
1answer
271 views
How do I convert an MDP with the reward function in the form $R(s,a,s')$ to and an MDP with a reward function in the form $R(s,a)$?
The AIMA book has an exercise about showing that an MDP with rewards of the form $r(s, a, s')$ can be converted to an MDP with rewards $r(s, a)$, and to an MDP with rewards $r(s)$ with equivalent ...
4
votes
1answer
71 views
How should I handle invalid actions in a grid world?
I'm building a really simple experiment, where I let an agent move from the bottom-left corner to the upper-right corner of a $3 \times 3$ grid world.
I plan to use DQN to do this. I'm having trouble ...
3
votes
1answer
78 views
Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?
I have implemented a simple Q-learning algorithm to minimize a cost function by setting the reward to the inverse of the cost of the action taken by the agent. The algorithm converges nicely, but ...
1
vote
1answer
65 views
What is the relationship between the reward function and the value function?
To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses ...
2
votes
1answer
88 views
How can I implement the reward function for an 8-DOF robot arm with TRPO?
I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to ...
10
votes
4answers
734 views
Counterexamples to the reward hypothesis
On Sutton and Barto's RL book, the reward hypothesis is stated as
that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative ...
2
votes
1answer
80 views
What is the difference between success rate and reward when dealing with binary and sparse rewards?
In OpenAI Gym "reward" is defined as:
reward (float): amount of reward achieved by the previous action. The
scale varies between environments, but the ...
2
votes
1answer
281 views
What is the optimal value function of the scaled version of the reward function?
Consider the reward function $r(s, a)$ with optimal state-action value function $q_*(s, a)$. What would be the optimal state-action value function of $c r(s, a)$, for $c \in \mathbb{R}$? Would it be $...
2
votes
1answer
143 views
How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?
Is there any empirical/theoretical evidence on the effect of initial values of state-action and state values on the training of an RL agent (the values an RL agent assigns to visited states) via MC ...
6
votes
2answers
552 views
How do we define the reward function for an environment?
How do you actually decide what reward value to give for each action in a given state for an environment?
Is this purely experimental and down to the programmer of the environment? So, is it a ...
1
vote
1answer
757 views
What is the reward system of reinforcement learning?
Can you describe this reward system in more detail?
I understand that the environment sends a signal indicating whether or not the action taken by the agent was 'good' or not, but it seems too simple.
...
4
votes
1answer
301 views
Can the rewards be stochastic when the transition model is deterministic?
Suppose we have a deterministic environment where knowing $s,a$ determines $s'$. Is it possible to get two different rewards $r\neq r'$ in some state $s_{\text{fixed}}$? Assume that $s_{\text{fixed}}$ ...
5
votes
2answers
245 views
How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?
In this video, the lecturer states that $R(s)$, $R(s, a)$ and $R(s, a, s')$ are equivalent representations of the reward function. Intuitively, this is the case, according to the same lecturer, ...
2
votes
1answer
53 views
Why does potential-based reward shaping seem to alter the optimal policy in this case?
It is known that every potential function won't alter the optimal policy [1]. I lack of understanding why is that.
The definition:
$$R' = R + F,$$ with $$F = \gamma\Phi(s') - \Phi(s),$$
where, let's ...
2
votes
2answers
130 views
How to define a reward function in POMDPs?
How do I define a reward function for my POMDP model?
In the literature, it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this ...
3
votes
1answer
113 views
Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true?
I am referring to eq. 3.6 (page 49) based on Sutton's online book and can be found in an image below.
I could not make sense of the final derivation of the equation $r(s, a, s')$. My question is ...
8
votes
1answer
2k views
What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?
I created an OpenAI Gym environment, and I would like to check the performance of the agent from OpenAI Baselines DQN approach on it.
In my environment, the best possible outcome for the agent is 0 - ...
