Questions tagged [reward-functions]
For questions about rewards functions (e.g. in the context of reinforcement learning, which may be denoted as $R(s, a)$).
65
questions
1
vote
0
answers
20
views
If we have a working reward function, would adding another action have a significant effect on the agent performance if task remains the same?
If we have a working reward function, providing the desired behavior and optimal policy in a continuous action/state-space problem, would adding another action significantly affect the possible ...
- 11
2
votes
1
answer
67
views
How to construct a reward function for a "wait and see" problem
I'm working on a problem that I think could probably be represented as a reinforcement learning task, but I'm uncertain about how to design the reward function. The core task is essentially a ...
- 21
1
vote
1
answer
52
views
How should I write the reward function to teach the agent the rules of this card game?
I'm quite new to reinforcement learning. I've been training the model for the following problem but the mean reward is stuck.
In a 5 by 5 board, each position can contain a card with a color (0-4) ...
- 111
1
vote
0
answers
41
views
What is commonly done for standardization/normalization of the targets in Deep Q-Learning?
I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
- 23
0
votes
1
answer
31
views
Does $R_{s}=E[R_{t}|S_{t}=s]$ indicate the reward we might expect on getting on average moving from any other state to $s$?
I'm trying to understand correctly what each "variable" in RL is and I'm not sure about $R_{s}$ the reward function. I used to think that it's the reward we may expect on average after ...
- 489
1
vote
0
answers
29
views
Normalisation of reward function
Problem
Currently, I have some problems defining a reward function for my RL project and mainly with how to normalise the score such that the highest possible score for all instances of the ...
- 21
1
vote
0
answers
38
views
How to teach Machine Learning Agent to destroy replicating objects in a puzzle game?
I have an unusual but very interesting problem. I have a game that is very similar to Toon Blast (a puzzle mobile game). It's based on a Match-2 mechanic in which you can destroy 2 or more connected ...
- 153
2
votes
1
answer
51
views
Is there a mathematical formalism to deal with a missing reward signal?
Typically, a Reinforcement Learning learning problem is formalized as finding an optimal policy for a Markov Decision Process (MDP). In many real-life situations, however, an agent can only get ...
- 163
-1
votes
1
answer
106
views
Is it a bad practice to use cumulative rewards in reinforcement learning
I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over ...
- 49
1
vote
1
answer
240
views
How should I define the reward function for a stock trading-like game?
Problem setting
Consider a game like trading a stock
At each step, the agent can buy / sell a stock.
Trade is a pair of ...
- 19
2
votes
1
answer
127
views
How to encourage the reinforcement-learning agent to reach the goal as quickly as possible, and what's the effect of discount factor?
I am trying to use reinforcement learning to solve a task and compare its performance to humans.
The task is to find a single target in a fixed number of locations. At each step, the agent will pick ...
- 173
1
vote
1
answer
75
views
How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?
Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is ...
- 121
1
vote
0
answers
42
views
Are there any deep RL algorithms that work well on finite MDPs and non-trivial terminal rewards?
I notice that most Deep Reinforcement Learning (DRL) works focus on Markov Decision Process (MDP) with an infinite time horizon.
Are there any algorithms that work well on finite MDP and non-trivial ...
- 111
1
vote
1
answer
91
views
How are rewards calculated for episodic tasks like playing chess or tic-tac-toe?
I am new to Reinforcement Learning and trying to understand the concept of reaping rewards during episodic tasks. I think in games like tic-tac-toe, rewards will be in terms of a win or lose. But does ...
- 11
1
vote
1
answer
69
views
How are these two versions of the Bellman optimality equation related?
I saw two versions of the optimality equation for $V_{*}(s)$ and $Q_{*}(s,a)$.
The first one is:
$$
V_{*}(s)=\max _{a} \sum_{s^{\prime}} P_{s s^{\prime}}^{a}\left(r(s, a)+\gamma V_{*}\left(s^{\prime}\...
- 49
1
vote
1
answer
70
views
In addition to the reward function, which other functions do I need to implement Q-learning?
In general, $Q$ function is defined as
$$Q : S \times A \rightarrow \mathbb{R}$$
$$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma \max\limits_{a} Q(s_{t+1},a) - Q(s_t,a_t)] $$
$\alpha$ and $\gamma$...
- 187
0
votes
1
answer
52
views
How would you shape a reward function if there was four quantities to optimize?
I found this article quite useful on how to shape a reward function in RL. However, the example they gave is quite simple, where the goal is to minimize only two quantities (velocity and distance).
...
- 36
0
votes
1
answer
101
views
How do we get the value of this state of an MDP, at time-step $h-2$, using dynamic programming?
I am trying to understand the problem below, represented as an MDP with four states (PU, PF, RU, and RF) and two actions (AS).
Let's consider V(RF), the value of the state RF. At time-step $h$, V(RF) ...
- 61
1
vote
1
answer
110
views
What happens with policy gradient methods if rewards are differentiable?
I would like some help with understanding why there is no explicit flow of information from the reward gradient to the parameters of the policy in policy gradient methods.
What I mean is the following,...
1
vote
1
answer
149
views
How to scale all positive continuous reward?
My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between ...
2
votes
1
answer
137
views
Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward
In the appendix of the Constrained Policy Optimization (CPO) paper (Arxiv), the authors denote the discounted future state distribution $d^\pi$ as:
$$d^\pi(s) = (1-\gamma) \sum_{t=0}^\infty{\gamma^t P(...
- 205
5
votes
1
answer
227
views
How to improve the reward signal when the rewards are sparse?
In cases where the reward is delayed, this can negatively impact a models ability to do proper credit assignment. In the case of a sparse reward, are there ways in which this can be negated?
In a ...
- 385
2
votes
1
answer
72
views
Can the rewards be matrices when using DQN?
I have a basic question. I'm working towards developing a reward function for my DQN. I'd like to train an RL agent to edit pixels on an image. I understand that convolutions are ideal for working ...
- 323
2
votes
1
answer
88
views
How can I go from $R(s)$ to $R(s,a)$ in this specific MDP?
I'm trying to implement a research paper, as explained in this other post, here the author of the paper assumed R as a function of both states and actions, while the code (and the MDP) I'm using to ...
2
votes
0
answers
37
views
Is better to reward short- or long-term progress in Q-learning?
I have been training some kind of agent to reach a target using a Q-learning based approach, and I have tried two different types of rewards:
Long-term reward: $\mathrm{reward} = - \mathrm{distance}(\...
- 1,087
0
votes
0
answers
99
views
Is my reward function non-Markovian?
I am working on an RL problem where the time when the agent obtains the reward for taking action $a$ in time step $t$ is stochastic. In fact, there is no immediate reward for taking action $a$ in time ...
1
vote
1
answer
60
views
If the reward function of an environment depends on some initial conditions, should I create a separate environment for each condition?
I would like some guidance on how to design an Environment for a Reinforcement Learning agent where the stopping conditions and rewards for the environment change based on an initial set of input ...
- 15
0
votes
0
answers
42
views
Thompson sampling with Bernoulli prior and non-binary reward update
I am solving a problem for which I have to select the best possible servers (level 1) to hit for a given data. These servers (level 1) in turn hit some other servers (level 2) to complete the request. ...
2
votes
0
answers
26
views
How can I discourage the RL agent from drawing in a zero-sum game?
My agent receives $1, 0, -1$ rewards for winning, drawing, and losing the game, respectively. What would be the consequences of setting reward to $-1$ for draws? Would that encourage the agent to win ...
- 663
6
votes
1
answer
501
views
Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?
If we shift the rewards by any constant (which is a type of reward shaping), the optimal state-action value function (and so optimal policy) does not change. The proof of this fact can be found here.
...
- 33.1k
2
votes
2
answers
275
views
How should I define the reward function to solve the Wumpus game with deep Q-learning?
I'm writing a DQN agent for the Wumpus game.
Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...
2
votes
1
answer
199
views
What are proxy reward functions?
The understanding I have is that they somehow adjust the objective to make it easier to meet, without changing the reward function.
... the observed proxy reward function is the approximate solution ...
- 517
3
votes
2
answers
312
views
How to apply Q-learning when rewards is only available at the last state?
I have a scheduling problem in which there are $n$ slots and $m$ clients. I am trying to solve the problem using Q-learning so I have made the following state-action model.
A state $s_t$ is given by ...
- 247
3
votes
1
answer
1k
views
What are the pros and cons of sparse and dense rewards in reinforcement learning?
From what I understand, if the rewards are sparse the agent will have to explore more to get rewards and learn the optimal policy, whereas if the rewards are dense in time, the agent is quickly guided ...
1
vote
0
answers
86
views
Given the daily stock prices of the last 3 years, how should I sample the training data for episodic RL?
I am playing around with a stock trading agent trained via (deep) reinforcement learning, including memory replay. The agent is trained for 1000 episodes, where each episode consists of 180 timesteps (...
- 111
2
votes
0
answers
52
views
How to combine two differently equally important signals into the reward function, that have different scales?
I have two signals that I want to use to model my reward.
The first one is the CPU TIME: running mean from this diagram:
The second one is the MAX RESIDUAL from this diagram:
Since they are both ...
- 352
12
votes
3
answers
815
views
Why is the reward in reinforcement learning always a scalar?
I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
- 223
5
votes
2
answers
2k
views
What are some best practices when trying to design a reward function?
Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward ...
3
votes
1
answer
735
views
Is a reward given at every step or only given when the RL agent fails or succeeds?
In reinforcement learning, an agent can receive a positive reward for correct actions and a negative reward for wrong actions, but does the agent also receive rewards for every other step/action?
- 1,153
3
votes
2
answers
96
views
Can rewards be decomposed into components?
I'm training a robot to walk to a specific $(x, y)$ point using TD3, and, for simplicity, I have something like ...
- 35
4
votes
1
answer
596
views
Why does shifting all the rewards have a different impact on the performance of the agent?
I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
- 41
5
votes
1
answer
517
views
How do I convert an MDP with the reward function in the form $R(s,a,s')$ to and an MDP with a reward function in the form $R(s,a)$?
The AIMA book has an exercise about showing that an MDP with rewards of the form $r(s, a, s')$ can be converted to an MDP with rewards $r(s, a)$, and to an MDP with rewards $r(s)$ with equivalent ...
- 436
5
votes
1
answer
262
views
How should I handle invalid actions in a grid world?
I'm building a really simple experiment, where I let an agent move from the bottom-left corner to the upper-right corner of a $3 \times 3$ grid world.
I plan to use DQN to do this. I'm having trouble ...
- 187
3
votes
1
answer
221
views
Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?
I have implemented a simple Q-learning algorithm to minimize a cost function by setting the reward to the inverse of the cost of the action taken by the agent. The algorithm converges nicely, but ...
- 77
1
vote
1
answer
115
views
Which reward function works for recommendation systems using knowledge graphs?
I've been reading this paper on recommendation systems using reinforcement learning (RL) and knowledge graphs (KGs).
To give some background, the graph has several (finitely many) entities, of which ...
4
votes
2
answers
484
views
Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?
Sutton and Barto define the state–action–next-state reward function, $r(s, a, s')$, as follows (equation 3.6, p. 49)
$$
r(s, a, s^{\prime}) \doteq \mathbb{E}\left[R_{t} \mid S_{t-1}=s, A_{t-1}=a, S_{t}...
- 127
1
vote
1
answer
315
views
What is the relationship between the reward function and the value function?
To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses ...
- 573
2
votes
1
answer
146
views
How can I implement the reward function for an 8-DOF robot arm with TRPO?
I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to ...
- 21
3
votes
3
answers
428
views
Is the policy really invariant under affine transformations of the reward function?
In the context of a Markov decision process, this paper says
it is well-known that the optimal policy is invariant to positive affine transformation of the reward function
On the other hand, ...
- 171
12
votes
4
answers
1k
views
Counterexamples to the reward hypothesis
On Sutton and Barto's RL book, the reward hypothesis is stated as
that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative ...
- 221