Newest 'reward-functions' Questions

1

vote

1answer

62 views

In addition to the reward function, which other functions do I need to implement Q-learning?

In general, $Q$ function is defined as $$Q : S \times A \rightarrow \mathbb{R}$$ $$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma \max\limits_{a} Q(s_{t+1},a) - Q(s_t,a_t)] $$ $\alpha$ and $\gamma$...

asked Jun 28 at 14:50

satya

1876 bronze badges

0

votes

1answer

34 views

How would you shape a reward function if there was four quantities to optimize?

I found this article quite useful on how to shape a reward function in RL. However, the example they gave is quite simple, where the goal is to minimize only two quantities (velocity and distance). ...

reinforcement-learning deep-rl reward-functions reward-design reward-shaping

asked Jun 17 at 8:01

BAKYAC

364 bronze badges

0

votes

1answer

94 views

How do we get the value of this state of an MDP, at time-step $h-2$, using dynamic programming?

I am trying to understand the problem below, represented as an MDP with four states (PU, PF, RU, and RF) and two actions (AS). Let's consider V(RF), the value of the state RF. At time-step $h$, V(RF) ...

markov-decision-process reward-functions bellman-equations policy-iteration dynamic-programming

asked Jun 3 at 4:37

AHF

615 bronze badges

1

vote

1answer

58 views

What happens with policy gradient methods if rewards are differentiable?

I would like some help with understanding why there is no explicit flow of information from the reward gradient to the parameters of the policy in policy gradient methods. What I mean is the following,...

reinforcement-learning deep-rl reward-functions

asked May 19 at 13:48

External Supplier Staff

111 bronze badge

1

vote

1answer

52 views

How to scale all positive continuous reward?

My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between ...

reinforcement-learning q-learning reward-functions normalisation standardisation

asked Apr 26 at 21:06

fardis nadimi

191 bronze badge

2

votes

1answer

66 views

Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward

In the appendix of the Constrained Policy Optimization (CPO) paper (Arxiv), the authors denote the discounted future state distribution $d^\pi$ as: $$d^\pi(s) = (1-\gamma) \sum_{t=0}^\infty{\gamma^t P(...

reinforcement-learning reward-functions constrained-optimization

asked Mar 4 at 12:47

jazzchipc

1855 bronze badges

5

votes

1answer

128 views

How to improve the reward signal when the rewards are sparse?

In cases where the reward is delayed, this can negatively impact a models ability to do proper credit assignment. In the case of a sparse reward, are there ways in which this can be negated? In a ...

reinforcement-learning reward-functions sparse-rewards delayed-rewards potential-reward-shaping

asked Feb 3 at 18:17

tryingtolearn

3751 gold badge2 silver badges9 bronze badges

2

votes

1answer

53 views

Can the rewards be matrices when using DQN?

I have a basic question. I'm working towards developing a reward function for my DQN. I'd like to train an RL agent to edit pixels on an image. I understand that convolutions are ideal for working ...

reinforcement-learning dqn reward-functions reward-design multi-objective-rl

asked Feb 2 at 12:57

junfanbl

2931 silver badge6 bronze badges

2

votes

1answer

64 views

How can I go from $R(s)$ to $R(s,a)$ in this specific MDP?

I'm trying to implement a research paper, as explained in this other post, here the author of the paper assumed R as a function of both states and actions, while the code (and the MDP) I'm using to ...

reinforcement-learning markov-decision-process reward-functions

asked Jan 27 at 14:54

ИванКарамазов

415 bronze badges

2

votes

1answer

189 views

What is the difference between a fitness function and a reward function?

In reinforcement learning (RL), the reward function (RF), which can be denoted as $r(s)$, $r(s, a)$, $r(s, a, s')$, $r(s, s')$ depending on its specific definition, provides the learning signal, which ...

reinforcement-learning objective-functions evolutionary-algorithms reward-functions fitness-functions

asked Jan 20 at 14:50

nbro♦

28.2k6 gold badges56 silver badges122 bronze badges

2

votes

0answers

28 views

Is better to reward short- or long-term progress in Q-learning?

I have been training some kind of agent to reach a target using a Q-learning based approach, and I have tried two different types of rewards: Long-term reward: $\mathrm{reward} = - \mathrm{distance}(\...

reinforcement-learning q-learning dqn reward-design reward-functions

asked Jan 18 at 13:32

Thomas Wagenaar

1,0175 silver badges10 bronze badges

0

votes

0answers

62 views

Is my reward function non-Markovian?

I am working on an RL problem where the time when the agent obtains the reward for taking action $a$ in time step $t$ is stochastic. In fact, there is no immediate reward for taking action $a$ in time ...

reinforcement-learning terminology reference-request reward-functions markov-property

asked Jan 10 at 22:10

shirin elahi

1

vote

1answer

51 views

If the reward function of an environment depends on some initial conditions, should I create a separate environment for each condition?

I would like some guidance on how to design an Environment for a Reinforcement Learning agent where the stopping conditions and rewards for the environment change based on an initial set of input ...

reinforcement-learning environment reward-design reward-functions

asked Nov 24 '20 at 15:46

RL_NOOB

153 bronze badges

0

votes

0answers

37 views

Thompson sampling with Bernoulli prior and non-binary reward update

I am solving a problem for which I have to select the best possible servers (level 1) to hit for a given data. These servers (level 1) in turn hit some other servers (level 2) to complete the request. ...

reinforcement-learning reward-design reward-functions bayesian-statistics thompson-sampling

asked Nov 16 '20 at 7:56

PUNEET AGARWAL

1

2

votes

0answers

22 views

How can I discourage the RL agent from drawing in a zero-sum game?

My agent receives $1, 0, -1$ rewards for winning, drawing, and losing the game, respectively. What would be the consequences of setting reward to $-1$ for draws? Would that encourage the agent to win ...

reinforcement-learning reward-design reward-functions zero-sum-games

asked Nov 15 '20 at 22:33

mark mark

6332 silver badges18 bronze badges

6

votes

1answer

266 views

Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?

If we shift the rewards by any constant (which is a type of reward shaping), the optimal state-action value function (and so optimal policy) does not change. The proof of this fact can be found here. ...

reinforcement-learning proofs reward-shaping reward-functions

asked Nov 1 '20 at 23:09

nbro♦

28.2k6 gold badges56 silver badges122 bronze badges

0

votes

1answer

121 views

What is the optimal value function of the shifted version of the reward function?

Similarly to this question that I asked some time ago, what is the optimal value function of the shifted (by some constant $c$) version of some reward function? More precisely, let's assume that $r(s, ...

reinforcement-learning proofs bellman-equations reward-shaping reward-functions

asked Nov 1 '20 at 0:28

nbro♦

28.2k6 gold badges56 silver badges122 bronze badges

2

votes

2answers

169 views

How should I define the reward function to solve the Wumpus game with deep Q-learning?

I'm writing a DQN agent for the Wumpus game. Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...

reinforcement-learning dqn deep-rl reward-design reward-functions

asked Oct 20 '20 at 7:40

Edwin Carlsson

212 bronze badges

2

votes

1answer

129 views

What are proxy reward functions?

The understanding I have is that they somehow adjust the objective to make it easier to meet, without changing the reward function. ... the observed proxy reward function is the approximate solution ...

reinforcement-learning definitions papers reward-functions

asked Sep 20 '20 at 14:53

mugoh

4951 silver badge17 bronze badges

3

votes

2answers

151 views

How to apply Q-learning when rewards is only available at the last state?

I have a scheduling problem in which there are $n$ slots and $m$ clients. I am trying to solve the problem using Q-learning so I have made the following state-action model. A state $s_t$ is given by ...

reinforcement-learning q-learning reward-functions sparse-rewards combinatorial-optimization

asked Aug 27 '20 at 14:17

zdm

2471 silver badge6 bronze badges

3

votes

1answer

619 views

What are the pros and cons of sparse and dense rewards in reinforcement learning?

From what I understand, if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided ...

reinforcement-learning comparison reward-functions sparse-rewards dense-rewards

asked Aug 13 '20 at 7:05

epsilon-emperor

8901 silver badge13 bronze badges

2

votes

0answers

40 views

How to combine two differently equally important signals into the reward function, that have different scales?

I have two signals that I want to use to model my reward. The first one is the CPU TIME: running mean from this diagram: The second one is the MAX RESIDUAL from this diagram: Since they are both ...

reinforcement-learning deep-rl reward-design reward-functions

asked Aug 7 '20 at 11:48

tmaric

3321 silver badge8 bronze badges

10

votes

3answers

463 views

Why is the reward in reinforcement learning always a scalar?

I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...

reinforcement-learning rewards reward-functions multi-objective-rl

asked Aug 6 '20 at 22:06

Sid Mani

2033 bronze badges

4

votes

2answers

430 views

What are some best practices when trying to design a reward function?

Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward ...

reinforcement-learning reward-design reward-functions reward-shaping inverse-rl

asked Aug 3 '20 at 16:30

12 rhombi in grid w no corners

1555 bronze badges

3

votes

1answer

369 views

Is a reward given at every step or only given when the RL agent fails or succeeds?

In reinforcement learning, an agent can receive a positive reward for correct actions and a negative reward for wrong actions, but does the agent also receive rewards for every other step/action?

reinforcement-learning reward-design reward-functions reward-shaping dense-rewards

asked Jul 21 '20 at 9:59

datdinhquoc

1,0406 silver badges24 bronze badges

3

votes

2answers

90 views

Can rewards be decomposed into components?

I'm training a robot to walk to a specific $(x, y)$ point using TD3, and, for simplicity, I have something like ...

reinforcement-learning reward-design reward-functions multi-objective-rl reward-hypothesis

asked Jul 9 '20 at 16:49

pinkie pAI

353 bronze badges

4

votes

1answer

309 views

Why does shifting all the rewards have a different impact on the performance of the agent?

I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...

reinforcement-learning dqn rewards reward-shaping reward-functions

asked Jul 1 '20 at 1:57

Fishfish

411 bronze badge

5

votes

1answer

290 views

How do I convert an MDP with the reward function in the form $R(s,a,s')$ to and an MDP with a reward function in the form $R(s,a)$?

The AIMA book has an exercise about showing that an MDP with rewards of the form $r(s, a, s')$ can be converted to an MDP with rewards $r(s, a)$, and to an MDP with rewards $r(s)$ with equivalent ...

reinforcement-learning markov-decision-process proofs reward-functions

asked May 25 '20 at 11:19

Asher

4262 silver badges8 bronze badges

5

votes

1answer

95 views

How should I handle invalid actions in a grid world?

I'm building a really simple experiment, where I let an agent move from the bottom-left corner to the upper-right corner of a $3 \times 3$ grid world. I plan to use DQN to do this. I'm having trouble ...

reinforcement-learning q-learning dqn reward-design reward-functions

asked May 19 '20 at 3:39

o_yeah

1874 bronze badges

3

votes

1answer

110 views

Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

I have implemented a simple Q-learning algorithm to minimize a cost function by setting the reward to the inverse of the cost of the action taken by the agent. The algorithm converges nicely, but ...

reinforcement-learning q-learning rewards reward-design reward-functions

asked May 14 '20 at 9:01

EArwa

774 bronze badges

1

vote

1answer

70 views

What is the relationship between the reward function and the value function?

To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses ...

reinforcement-learning comparison value-functions reward-functions

asked Apr 3 '20 at 22:06

mason7663

5731 silver badge7 bronze badges

2

votes

1answer

96 views

How can I implement the reward function for an 8-DOF robot arm with TRPO?

I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to ...

reinforcement-learning gym reward-design reward-functions trust-region-policy-optimization

asked Mar 5 '20 at 2:10

user1690356

211 bronze badge

11

votes

4answers

915 views

Counterexamples to the reward hypothesis

On Sutton and Barto's RL book, the reward hypothesis is stated as that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative ...

reinforcement-learning sutton-barto reward-design reward-functions reward-hypothesis

asked Nov 19 '19 at 2:30

Bananin

2111 silver badge3 bronze badges

2

votes

1answer

89 views

What is the difference between success rate and reward when dealing with binary and sparse rewards?

In OpenAI Gym "reward" is defined as: reward (float): amount of reward achieved by the previous action. The scale varies between environments, but the ...

reinforcement-learning terminology papers reward-functions hindsight-experience-replay

asked Oct 30 '19 at 12:09

rrz0

2331 silver badge6 bronze badges

2

votes

1answer

296 views

What is the optimal value function of the scaled version of the reward function?

Consider the reward function $r(s, a)$ with optimal state-action value function $q_*(s, a)$. What would be the optimal state-action value function of $c r(s, a)$, for $c \in \mathbb{R}$? Would it be $...

reinforcement-learning proofs bellman-equations reward-shaping reward-functions

asked Sep 15 '19 at 20:58

nbro♦

28.2k6 gold badges56 silver badges122 bronze badges

2

votes

1answer

216 views

How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?

Is there any empirical/theoretical evidence on the effect of initial values of state-action and state values on the training of an RL agent (the values an RL agent assigns to visited states) via MC ...

reinforcement-learning value-functions reward-design reward-functions weights-initialization

asked Jul 6 '19 at 20:47

user9947

7

votes

2answers

687 views

How do we define the reward function for an environment?

How do you actually decide what reward value to give for each action in a given state for an environment? Is this purely experimental and down to the programmer of the environment? So, is it a ...

reinforcement-learning reward-functions reward-design reward-shaping

asked May 12 '19 at 0:15

Hazzaldo

2411 silver badge6 bronze badges

1

vote

1answer

869 views

What is the reward system of reinforcement learning?

Can you describe this reward system in more detail? I understand that the environment sends a signal indicating whether or not the action taken by the agent was 'good' or not, but it seems too simple. ...

reinforcement-learning markov-decision-process rewards reward-functions

asked May 10 '19 at 23:43

12 rhombi in grid w no corners

1555 bronze badges

4

votes

1answer

321 views

Can the rewards be stochastic when the transition model is deterministic?

Suppose we have a deterministic environment where knowing $s,a$ determines $s'$. Is it possible to get two different rewards $r\neq r'$ in some state $s_{\text{fixed}}$? Assume that $s_{\text{fixed}}$ ...

reinforcement-learning markov-decision-process reward-design reward-functions

asked Feb 12 '19 at 10:33

rewardbabe

411 bronze badge

5

votes

2answers

271 views

How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?

In this video, the lecturer states that $R(s)$, $R(s, a)$ and $R(s, a, s')$ are equivalent representations of the reward function. Intuitively, this is the case, according to the same lecturer, ...

reinforcement-learning markov-decision-process proofs notation reward-functions

asked Feb 7 '19 at 15:38

nbro♦

28.2k6 gold badges56 silver badges122 bronze badges

2

votes

1answer

57 views

Why does potential-based reward shaping seem to alter the optimal policy in this case?

It is known that every potential function won't alter the optimal policy [1]. I lack of understanding why is that. The definition: $$R' = R + F,$$ with $$F = \gamma\Phi(s') - \Phi(s),$$ where, let's ...

reinforcement-learning reward-functions reward-shaping potential-reward-shaping

asked Jan 26 '19 at 14:11

ScientiaEtVeritas

1555 bronze badges

2

votes

2answers

133 views

How to define a reward function in POMDPs?

How do I define a reward function for my POMDP model? In the literature, it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this ...

reinforcement-learning reward-design reward-functions pomdp

asked Oct 29 '18 at 16:09

Bryan McGill

3612 silver badges11 bronze badges

3

votes

1answer

114 views

Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true?

I am referring to eq. 3.6 (page 49) based on Sutton's online book and can be found in an image below. I could not make sense of the final derivation of the equation $r(s, a, s')$. My question is ...

reinforcement-learning math notation reward-functions probability-theory

asked Oct 19 '18 at 9:34

alfa_80

1333 bronze badges

8

votes

1answer

2k views

What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?

I created an OpenAI Gym environment, and I would like to check the performance of the agent from OpenAI Baselines DQN approach on it. In my environment, the best possible outcome for the agent is 0 - ...

reinforcement-learning q-learning dqn implementation reward-functions

asked May 29 '17 at 9:02

AlexGuevara

2631 silver badge8 bronze badges

Questions tagged [reward-functions]

Related Tags