Questions tagged [reward-functions]

For questions about rewards functions (e.g. in the context of reinforcement learning, which may be denoted as $R(s, a)$).

61 questions
Filter by
Sorted by
Tagged with
662 views

Why is the reward in reinforcement learning always a scalar?

I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
1k views

Counterexamples to the reward hypothesis

On Sutton and Barto's RL book, the reward hypothesis is stated as that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative ...
2k views

What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?

I created an OpenAI Gym environment, and I would like to check the performance of the agent from OpenAI Baselines DQN approach on it. In my environment, the best possible outcome for the agent is 0 - ...
3k views

I am working to build a deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are ...
868 views

How do we define the reward function for an environment?

How do you actually decide what reward value to give for each action in a given state for an environment? Is this purely experimental and down to the programmer of the environment? So, is it a ...
471 views

Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?

If we shift the rewards by any constant (which is a type of reward shaping), the optimal state-action value function (and so optimal policy) does not change. The proof of this fact can be found here. ...
415 views

How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?

In this video, the lecturer states that $R(s)$, $R(s, a)$ and $R(s, a, s')$ are equivalent representations of the reward function. Intuitively, this is the case, according to the same lecturer, ...
173 views

How to improve the reward signal when the rewards are sparse?

In cases where the reward is delayed, this can negatively impact a models ability to do proper credit assignment. In the case of a sparse reward, are there ways in which this can be negated? In a ...
415 views

How do I convert an MDP with the reward function in the form $R(s,a,s')$ to and an MDP with a reward function in the form $R(s,a)$?

The AIMA book has an exercise about showing that an MDP with rewards of the form $r(s, a, s')$ can be converted to an MDP with rewards $r(s, a)$, and to an MDP with rewards $r(s)$ with equivalent ...
159 views

How should I handle invalid actions in a grid world?

I'm building a really simple experiment, where I let an agent move from the bottom-left corner to the upper-right corner of a $3 \times 3$ grid world. I plan to use DQN to do this. I'm having trouble ...
1k views

What are some best practices when trying to design a reward function?

Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward ...
1k views

How can I ensure convergence of DDQN, if the true Q-values for different actions in the same state are very close?

I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future ...
369 views

Can the rewards be stochastic when the transition model is deterministic?

Suppose we have a deterministic environment where knowing $s,a$ determines $s'$. Is it possible to get two different rewards $r\neq r'$ in some state $s_{\text{fixed}}$? Assume that $s_{\text{fixed}}$ ...
462 views

Why does shifting all the rewards have a different impact on the performance of the agent?

I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
66 views

How define a reward function for a humanoid agent whose goal is to stand up from the ground?

I'm trying to teach a humanoid agent how to stand up after falling. The episode starts with the agent lying on the floor with its back touching the ground, and its goal is to stand up in the shortest ...
53 views

What research has been done on learning non-Markovian reward functions?

Recently, some work has been done planning and learning in Non-Markovian Decision Processes, that is, decision-making with temporally extended rewards. In these settings, a particular reward is ...
94 views

Can rewards be decomposed into components?

I'm training a robot to walk to a specific $(x, y)$ point using TD3, and, for simplicity, I have something like ...
218 views

How to apply Q-learning when rewards is only available at the last state?

I have a scheduling problem in which there are $n$ slots and $m$ clients. I am trying to solve the problem using Q-learning so I have made the following state-action model. A state $s_t$ is given by ...
162 views

Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

I have implemented a simple Q-learning algorithm to minimize a cost function by setting the reward to the inverse of the cost of the action taken by the agent. The algorithm converges nicely, but ...
260 views

What is the difference between a fitness function and a reward function?

In reinforcement learning (RL), the reward function (RF), which can be denoted as $r(s)$, $r(s, a)$, $r(s, a, s')$, $r(s, s')$ depending on its specific definition, provides the learning signal, which ...
960 views

What are the pros and cons of sparse and dense rewards in reinforcement learning?

From what I understand, if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided ...
529 views

Is a reward given at every step or only given when the RL agent fails or succeeds?

In reinforcement learning, an agent can receive a positive reward for correct actions and a negative reward for wrong actions, but does the agent also receive rewards for every other step/action?
346 views

Is the policy really invariant under affine transformations of the reward function?

In the context of a Markov decision process, this paper says it is well-known that the optimal policy is invariant to positive affine transformation of the reward function On the other hand, ...
304 views

How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?

Is there any empirical/theoretical evidence on the effect of initial values of state-action and state values on the training of an RL agent (the values an RL agent assigns to visited states) via MC ...
118 views

Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true?

I am referring to eq. 3.6 (page 49) based on Sutton's online book and can be found in an image below. I could not make sense of the final derivation of the equation $r(s, a, s')$. My question is ...
86 views

54 views

If the reward function of an environment depends on some initial conditions, should I create a separate environment for each condition?

I would like some guidance on how to design an Environment for a Reinforcement Learning agent where the stopping conditions and rewards for the environment change based on an initial set of input ...
145 views

What is the relationship between the reward function and the value function?

To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses ...
A Markov Decision Process (MDP) is a mathematical model for sequential decision-making in stochastic environments. Formally, we can define an MDP as a tuple $M = (S, A, p, \gamma)$, where $S$ is the ...