Questions tagged [reward-functions]
For questions about rewards functions (e.g. in the context of reinforcement learning, which may be denoted as $R(s, a)$).
88
questions
0
votes
0
answers
3
views
name of Reward function that utilizes the rewards of the next n steps
I have a problem with continuous time, observation and action space. I am discretizing the time to be able to apply the usual Reinforcement Learning algorithms (I chose PPO). The problem consists of a ...
0
votes
1
answer
52
views
Why is R(s) more restrictive than R(s, a) in an MDP?
I am quite new to RL. I would like to know why a state-dependent reward function R(s) is more restrictive than a state-action-dependent reward function R(s, a)? And why is it that a policy can be ...
- 39.7k
2
votes
1
answer
351
views
How can I implement the reward function for an 8-DOF robot arm with TRPO?
I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to ...
CommunityBot
- 1
1
vote
1
answer
597
views
How to scale all positive continuous reward?
My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between ...
CommunityBot
- 1
5
votes
1
answer
185
views
How define a reward function for a humanoid agent whose goal is to stand up from the ground?
I'm trying to teach a humanoid agent how to stand up after falling. The episode starts with the agent lying on the floor with its back touching the ground, and its goal is to stand up in the shortest ...
CommunityBot
- 1
0
votes
0
answers
46
views
How to handle penalty and reward occurring simultaneously
Assume the following scenario:
We have an agent that acts on an environment where the agent should never take an action that results in him leaving the environment. For example, imagine an agent ...
- 39.7k
7
votes
1
answer
3k
views
Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?
If we shift the rewards by any constant (which is a type of reward shaping), the optimal state-action value function (and so optimal policy) does not change. The proof of this fact can be found here.
...
- 39.7k
0
votes
0
answers
20
views
If the agent is at the same state but at different times and receives a different reward, wouldn't this be violating somehow the MDP assumption?
I've been trying to train an agent, I've received and read suggestions to improve its speed to reach the goal. The suggestion is to use a time penalty, for example, adding $-0.1$ to the reward each ...
- 39.7k
0
votes
1
answer
34
views
Could you explain the derivation of the expectation equation of equation 3.6 in Sutton & Barto? [duplicate]
I don't understand the last equality.
Here is my derivation
$r(s,a,s')=\sum_{r\in R} r p(r|s,a,s')=\sum_{r\in R} r \frac{p(s,a,s'|r)p(r)}{p(s,a,s')}$
Could you give me the correct steps to derive them?...
- 195
0
votes
1
answer
81
views
Can a reward function have various cases?
I'm doing a Q-learning algorithm and I'm designing my reward function. Basically I'm working on optimizing a network while changing some parameters. My metric to measure its optimization is the delay ...
- 3
6
votes
2
answers
2k
views
Why does shifting all the rewards have a different impact on the performance of the agent?
I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
- 11
1
vote
0
answers
28
views
What does "shuffle the comparisons into one dataset" mean?
I couldn't understand the wording here.
What does "shuffle the comparisons into one dataset" mean?
How does the method they use don't have $K \choose 2$ forward passes for K completions? Do ...
- 39.7k
0
votes
1
answer
116
views
Will my Q values keep going up forever?
In Q-learning,the q values can be updated by the bellman equation. What happens with my Q values is that they keep going up forever, in accordance with the more I train. After 10,000 training episodes,...
CommunityBot
- 1
1
vote
1
answer
57
views
How to setup a reinforcement learning model that changes the values of $x$ to maximize $y$ in $y = f(x)$?
Assuming a relation such that $y = f(x)$, where $y$ represents a scalar and $x \in 20 \times 1$ vector consisting of zeros and ones, I want to set up a reinforcement learning model that changes the ...
- 39.7k
2
votes
0
answers
70
views
In RL, is the quantification of the reward function arbitrary? Does it affect the learning?
There are different ways to set the reward function, such as extrinsic (externally provided rewards), intrinsic (the rewards are generated by the agents themselves based on their internal state and ...
- 21
1
vote
1
answer
172
views
Is there a reward function that would encourage exploration in this case?
I am new to Reinforcement Learning. I am trying to train PPO agent for citylearn. The goal is to lower two environmental variables from observations. The default reward function is
...
- 39.7k
1
vote
1
answer
184
views
Is it necessary to have a constant reward in the terminal state?
I have downloaded the grid world project form this link. I have executed the project multiple times using:
python gridworld.py -k 20 -a q -r -0.2 -s 90
I have ...
- 39.7k
1
vote
2
answers
2k
views
What is the relationship between the reward function and the value function?
To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses ...
- 121
1
vote
0
answers
42
views
Would the optimal policy remain same, if I replace R with V*?
In the context of RL, say I'm performing Value Iteration on a reward function R1. And the converged optimal policy is P1 and values are V1.
Then, let's say I set rewards to be R2=V1 and perform value ...
1
vote
1
answer
186
views
How to normalizing various elements of the reward function?
Suppose I have a reward function $R$ that I wish to penalize w.r.t two distinct phenomenons $A$ and $B$. $A$, for example, could represent the phenomenon of the state not crossing some boundary $[s_1,...
- 1,094
0
votes
0
answers
29
views
Bouding the state using the reward in RL
I'm wondering what the common approaches are bounding out state $s\in\mathbf{R}$ to some values $\in[s_0,s_1]$ is required. So in my case, for example, the state represents an angle of rotation, that ...
- 371
0
votes
0
answers
34
views
Reward Function for Reinforcement Learning model
I am trying to create a reinforcement learning model to control the acceleration of a car. I am designing the model such that initially the acceleration is provided and then deceleration is provided ...
2
votes
2
answers
526
views
How should I define the reward function to solve the Wumpus game with deep Q-learning?
I'm writing a DQN agent for the Wumpus game.
Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...
- 927
0
votes
0
answers
125
views
Shaping reward so that it maximizes multiple components together
I am fairly new to RL and I compete in AWS DeepRacer student league. The main task there is to create a reward function. All the hyperparameters and action space are fixed. So far, I know how to shape ...
3
votes
1
answer
136
views
Reward design or Inverse reinforcement learning?
I'm working on a reinforcement learning project where I only have demonstrations (i.e. set of states and actions). During my research on how handle the reward signal, I noticed that research papers ...
- 380
2
votes
1
answer
751
views
How to deal with small reward values
In my environment rewards are generally small, e.g. [-0.01, 0.01]. My concern is that small reward values might get dominated or distorted by the noise during the training. Does it make sense to scale ...
- 30.4k
2
votes
1
answer
670
views
What happens with policy gradient methods if rewards are differentiable?
I would like some help with understanding why there is no explicit flow of information from the reward gradient to the parameters of the policy in policy gradient methods.
What I mean is the following,...
0
votes
2
answers
140
views
Is there a way to form a reward function so that it would take into account the order of the actions?
I want to design a multi-arm bandit system for a multi-step, multi-location system. Locations are dynamic, so I can not design the system based on them. In each location, the alternative actions that ...
- 39.7k
0
votes
0
answers
213
views
How to solve a reinforcement learning problem with a stochastic reward function?
In a discrete time system, an environment has an unknown reward probability $p(r|s,a)$. However, the transition probability $p(s'\mid s,a)$ is deterministic.
In my case, the reward for the same action ...
- 39.7k
2
votes
0
answers
253
views
How to solve a reinforcement learning problem with changing rewards?
I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for ...
2
votes
1
answer
410
views
In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?
In the context of my problem, the "true" reward is not additive. Realistically, the more reward the agent has already accumulated, the easier it becomes to accumulate even more. That's to ...
- 342
1
vote
0
answers
25
views
If we have a working reward function, would adding another action have a significant effect on the agent performance if task remains the same?
If we have a working reward function, providing the desired behavior and optimal policy in a continuous action/state-space problem, would adding another action significantly affect the possible ...
- 39.7k
2
votes
1
answer
135
views
How to construct a reward function for a "wait and see" problem
I'm working on a problem that I think could probably be represented as a reinforcement learning task, but I'm uncertain about how to design the reward function. The core task is essentially a ...
- 39.7k
1
vote
1
answer
239
views
How should I write the reward function to teach the agent the rules of this card game?
I'm quite new to reinforcement learning. I've been training the model for the following problem but the mean reward is stuck.
In a 5 by 5 board, each position can contain a card with a color (0-4) ...
- 30.4k
1
vote
0
answers
662
views
What is commonly done for standardization/normalization of the targets in Deep Q-Learning?
I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
- 39.7k
2
votes
1
answer
176
views
Where are the parentheses in the definition of $r(s,a)$?
I am new to RL and I am trying to work through the book Reinforcement Learning: An Introduction I (Sutton & Barto, 2018). In chapter 3 on Finite Markov Decision Processes, the authors write the ...
- 39.7k
4
votes
2
answers
969
views
Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?
Sutton and Barto define the state–action–next-state reward function, $r(s, a, s')$, as follows (equation 3.6, p. 49)
$$
r(s, a, s^{\prime}) \doteq \mathbb{E}\left[R_{t} \mid S_{t-1}=s, A_{t-1}=a, S_{t}...
- 39.7k
0
votes
1
answer
53
views
Does $R_{s}=E[R_{t}|S_{t}=s]$ indicate the reward we might expect on getting on average moving from any other state to $s$?
I'm trying to understand correctly what each "variable" in RL is and I'm not sure about $R_{s}$ the reward function. I used to think that it's the reward we may expect on average after ...
- 39.7k
1
vote
0
answers
96
views
Normalisation of reward function
Problem
Currently, I have some problems defining a reward function for my RL project and mainly with how to normalise the score such that the highest possible score for all instances of the ...
- 31
1
vote
1
answer
364
views
How are rewards calculated for episodic tasks like playing chess or tic-tac-toe?
I am new to Reinforcement Learning and trying to understand the concept of reaping rewards during episodic tasks. I think in games like tic-tac-toe, rewards will be in terms of a win or lose. But does ...
CommunityBot
- 1
4
votes
1
answer
2k
views
How can I ensure convergence of DDQN, if the true Q-values for different actions in the same state are very close?
I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future ...
- 39.7k
1
vote
1
answer
242
views
Which reward function works for recommendation systems using knowledge graphs?
I've been reading this paper on recommendation systems using reinforcement learning (RL) and knowledge graphs (KGs).
To give some background, the graph has several (finitely many) entities, of which ...
- 39.7k
3
votes
3
answers
943
views
Is the policy really invariant under affine transformations of the reward function?
In the context of a Markov decision process, this paper says
it is well-known that the optimal policy is invariant to positive affine transformation of the reward function
On the other hand, ...
1
vote
1
answer
187
views
How are these two versions of the Bellman optimality equation related?
I saw two versions of the optimality equation for $V_{*}(s)$ and $Q_{*}(s,a)$.
The first one is:
$$
V_{*}(s)=\max _{a} \sum_{s^{\prime}} P_{s s^{\prime}}^{a}\left(r(s, a)+\gamma V_{*}\left(s^{\prime}\...
- 39.7k
1
vote
0
answers
83
views
Are there any deep RL algorithms that work well on finite MDPs and non-trivial terminal rewards?
I notice that most Deep Reinforcement Learning (DRL) works focus on Markov Decision Process (MDP) with an infinite time horizon.
Are there any algorithms that work well on finite MDP and non-trivial ...
- 39.7k
1
vote
0
answers
145
views
Given the daily stock prices of the last 3 years, how should I sample the training data for episodic RL?
I am playing around with a stock trading agent trained via (deep) reinforcement learning, including memory replay. The agent is trained for 1000 episodes, where each episode consists of 180 timesteps (...
- 39.7k
1
vote
0
answers
63
views
How to teach Machine Learning Agent to destroy replicating objects in a puzzle game?
I have an unusual but very interesting problem. I have a game that is very similar to Toon Blast (a puzzle mobile game). It's based on a Match-2 mechanic in which you can destroy 2 or more connected ...
- 39.7k
4
votes
0
answers
71
views
What research has been done on learning non-Markovian reward functions?
Recently, some work has been done planning and learning in Non-Markovian Decision Processes, that is, decision-making with temporally extended rewards. In these settings, a particular reward is ...
- 39.7k
8
votes
1
answer
6k
views
Suitable reward function for trading buy and sell orders
I am working to build a deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are ...
- 39.7k
1
vote
1
answer
1k
views
How should I define the reward function for a stock trading-like game?
Problem setting
Consider a game like trading a stock
At each step, the agent can buy / sell a stock.
Trade is a pair of ...
- 39.7k