Questions tagged [reward-functions]

For questions about rewards functions (e.g. in the context of reinforcement learning, which may be denoted as $R(s, a)$).

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

name of Reward function that utilizes the rewards of the next n steps

I have a problem with continuous time, observation and action space. I am discretizing the time to be able to apply the usual Reinforcement Learning algorithms (I chose PPO). The problem consists of a ...
Georg Schneeberger's user avatar
0 votes
1 answer
52 views

Why is R(s) more restrictive than R(s, a) in an MDP?

I am quite new to RL. I would like to know why a state-dependent reward function R(s) is more restrictive than a state-action-dependent reward function R(s, a)? And why is it that a policy can be ...
TicTacToemat's user avatar
0 votes
0 answers
46 views

How to handle penalty and reward occurring simultaneously

Assume the following scenario: We have an agent that acts on an environment where the agent should never take an action that results in him leaving the environment. For example, imagine an agent ...
kklaw's user avatar
  • 143
0 votes
0 answers
20 views

If the agent is at the same state but at different times and receives a different reward, wouldn't this be violating somehow the MDP assumption?

I've been trying to train an agent, I've received and read suggestions to improve its speed to reach the goal. The suggestion is to use a time penalty, for example, adding $-0.1$ to the reward each ...
Andrea Carolina Mora Lopez's user avatar
0 votes
1 answer
34 views

Could you explain the derivation of the expectation equation of equation 3.6 in Sutton & Barto? [duplicate]

I don't understand the last equality. Here is my derivation $r(s,a,s')=\sum_{r\in R} r p(r|s,a,s')=\sum_{r\in R} r \frac{p(s,a,s'|r)p(r)}{p(s,a,s')}$ Could you give me the correct steps to derive them?...
tesio's user avatar
  • 195
0 votes
1 answer
81 views

Can a reward function have various cases?

I'm doing a Q-learning algorithm and I'm designing my reward function. Basically I'm working on optimizing a network while changing some parameters. My metric to measure its optimization is the delay ...
AdrienF's user avatar
1 vote
0 answers
28 views

What does "shuffle the comparisons into one dataset" mean?

I couldn't understand the wording here. What does "shuffle the comparisons into one dataset" mean? How does the method they use don't have $K \choose 2$ forward passes for K completions? Do ...
ali batur karakullukcu's user avatar
1 vote
1 answer
57 views

How to setup a reinforcement learning model that changes the values of $x$ to maximize $y$ in $y = f(x)$?

Assuming a relation such that $y = f(x)$, where $y$ represents a scalar and $x \in 20 \times 1$ vector consisting of zeros and ones, I want to set up a reinforcement learning model that changes the ...
Fly's user avatar
  • 111
2 votes
0 answers
70 views

In RL, is the quantification of the reward function arbitrary? Does it affect the learning?

There are different ways to set the reward function, such as extrinsic (externally provided rewards), intrinsic (the rewards are generated by the agents themselves based on their internal state and ...
John Prada's user avatar
0 votes
1 answer
116 views

Will my Q values keep going up forever?

In Q-learning,the q values can be updated by the bellman equation. What happens with my Q values is that they keep going up forever, in accordance with the more I train. After 10,000 training episodes,...
Kyotiq's user avatar
  • 1
1 vote
1 answer
184 views

Is it necessary to have a constant reward in the terminal state?

I have downloaded the grid world project form this link. I have executed the project multiple times using: python gridworld.py -k 20 -a q -r -0.2 -s 90 I have ...
AAA's user avatar
  • 111
1 vote
0 answers
42 views

Would the optimal policy remain same, if I replace R with V*?

In the context of RL, say I'm performing Value Iteration on a reward function R1. And the converged optimal policy is P1 and values are V1. Then, let's say I set rewards to be R2=V1 and perform value ...
famishedrover's user avatar
1 vote
1 answer
172 views

Is there a reward function that would encourage exploration in this case?

I am new to Reinforcement Learning. I am trying to train PPO agent for citylearn. The goal is to lower two environmental variables from observations. The default reward function is ...
Sai Dinesh Pola's user avatar
1 vote
1 answer
186 views

How to normalizing various elements of the reward function?

Suppose I have a reward function $R$ that I wish to penalize w.r.t two distinct phenomenons $A$ and $B$. $A$, for example, could represent the phenomenon of the state not crossing some boundary $[s_1,...
Hadar Sharvit's user avatar
0 votes
0 answers
34 views

Reward Function for Reinforcement Learning model

I am trying to create a reinforcement learning model to control the acceleration of a car. I am designing the model such that initially the acceleration is provided and then deceleration is provided ...
Aditya Prakash's user avatar
0 votes
0 answers
29 views

Bouding the state using the reward in RL

I'm wondering what the common approaches are bounding out state $s\in\mathbf{R}$ to some values $\in[s_0,s_1]$ is required. So in my case, for example, the state represents an angle of rotation, that ...
Hadar Sharvit's user avatar
0 votes
0 answers
125 views

Shaping reward so that it maximizes multiple components together

I am fairly new to RL and I compete in AWS DeepRacer student league. The main task there is to create a reward function. All the hyperparameters and action space are fixed. So far, I know how to shape ...
aniketvp24's user avatar
3 votes
1 answer
136 views

Reward design or Inverse reinforcement learning?

I'm working on a reinforcement learning project where I only have demonstrations (i.e. set of states and actions). During my research on how handle the reward signal, I noticed that research papers ...
Eman.suradi's user avatar
2 votes
1 answer
751 views

How to deal with small reward values

In my environment rewards are generally small, e.g. [-0.01, 0.01]. My concern is that small reward values might get dominated or distorted by the noise during the training. Does it make sense to scale ...
Mika's user avatar
  • 341
0 votes
2 answers
140 views

Is there a way to form a reward function so that it would take into account the order of the actions?

I want to design a multi-arm bandit system for a multi-step, multi-location system. Locations are dynamic, so I can not design the system based on them. In each location, the alternative actions that ...
Ferda-Ozdemir-Sonmez's user avatar
0 votes
0 answers
213 views

How to solve a reinforcement learning problem with a stochastic reward function?

In a discrete time system, an environment has an unknown reward probability $p(r|s,a)$. However, the transition probability $p(s'\mid s,a)$ is deterministic. In my case, the reward for the same action ...
Zhenzhen Gong's user avatar
2 votes
0 answers
253 views

How to solve a reinforcement learning problem with changing rewards?

I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for ...
Zhenzhen Gong's user avatar
2 votes
1 answer
410 views

In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?

In the context of my problem, the "true" reward is not additive. Realistically, the more reward the agent has already accumulated, the easier it becomes to accumulate even more. That's to ...
Vladimir Belik's user avatar
1 vote
0 answers
25 views

If we have a working reward function, would adding another action have a significant effect on the agent performance if task remains the same?

If we have a working reward function, providing the desired behavior and optimal policy in a continuous action/state-space problem, would adding another action significantly affect the possible ...
Philori's user avatar
  • 11
2 votes
1 answer
135 views

How to construct a reward function for a "wait and see" problem

I'm working on a problem that I think could probably be represented as a reinforcement learning task, but I'm uncertain about how to design the reward function. The core task is essentially a ...
user336650's user avatar
1 vote
1 answer
239 views

How should I write the reward function to teach the agent the rules of this card game?

I'm quite new to reinforcement learning. I've been training the model for the following problem but the mean reward is stuck. In a 5 by 5 board, each position can contain a card with a color (0-4) ...
durianice's user avatar
  • 113
1 vote
0 answers
662 views

What is commonly done for standardization/normalization of the targets in Deep Q-Learning?

I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
Peter's user avatar
  • 55
0 votes
1 answer
53 views

Does $R_{s}=E[R_{t}|S_{t}=s]$ indicate the reward we might expect on getting on average moving from any other state to $s$?

I'm trying to understand correctly what each "variable" in RL is and I'm not sure about $R_{s}$ the reward function. I used to think that it's the reward we may expect on average after ...
Daviiid's user avatar
  • 573
1 vote
0 answers
96 views

Normalisation of reward function

Problem Currently, I have some problems defining a reward function for my RL project and mainly with how to normalise the score such that the highest possible score for all instances of the ...
Jesse's user avatar
  • 31
1 vote
0 answers
63 views

How to teach Machine Learning Agent to destroy replicating objects in a puzzle game?

I have an unusual but very interesting problem. I have a game that is very similar to Toon Blast (a puzzle mobile game). It's based on a Match-2 mechanic in which you can destroy 2 or more connected ...
Jacob's user avatar
  • 153
2 votes
1 answer
94 views

Is there a mathematical formalism to deal with a missing reward signal?

Typically, a Reinforcement Learning learning problem is formalized as finding an optimal policy for a Markov Decision Process (MDP). In many real-life situations, however, an agent can only get ...
Onil90's user avatar
  • 173
0 votes
1 answer
875 views

Is it a bad practice to use cumulative rewards in reinforcement learning

I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over ...
Leibniz's user avatar
  • 69
1 vote
1 answer
1k views

How should I define the reward function for a stock trading-like game?

Problem setting Consider a game like trading a stock At each step, the agent can buy / sell a stock. Trade is a pair of ...
not another narcissist's user avatar
2 votes
1 answer
1k views

How to encourage the reinforcement-learning agent to reach the goal as quickly as possible, and what's the effect of discount factor?

I am trying to use reinforcement learning to solve a task and compare its performance to humans. The task is to find a single target in a fixed number of locations. At each step, the agent will pick ...
Cloudy's user avatar
  • 213
1 vote
1 answer
456 views

How do I compute the value function when the reward is only at the end in the context of actor-critic algorithms?

Consider the actor-critic reinforcement learning setting (actor and critic parameterized by a neural network). The reward is given only at the end of the episode (or when there is a timeout there is ...
cerebrou's user avatar
  • 141
1 vote
0 answers
83 views

Are there any deep RL algorithms that work well on finite MDPs and non-trivial terminal rewards?

I notice that most Deep Reinforcement Learning (DRL) works focus on Markov Decision Process (MDP) with an infinite time horizon. Are there any algorithms that work well on finite MDP and non-trivial ...
Qinsheng Zhang's user avatar
1 vote
1 answer
364 views

How are rewards calculated for episodic tasks like playing chess or tic-tac-toe?

I am new to Reinforcement Learning and trying to understand the concept of reaping rewards during episodic tasks. I think in games like tic-tac-toe, rewards will be in terms of a win or lose. But does ...
Saily_Shah's user avatar
1 vote
1 answer
187 views

How are these two versions of the Bellman optimality equation related?

I saw two versions of the optimality equation for $V_{*}(s)$ and $Q_{*}(s,a)$. The first one is: $$ V_{*}(s)=\max _{a} \sum_{s^{\prime}} P_{s s^{\prime}}^{a}\left(r(s, a)+\gamma V_{*}\left(s^{\prime}\...
Kronic's user avatar
  • 49
1 vote
1 answer
90 views

In addition to the reward function, which other functions do I need to implement Q-learning?

In general, $Q$ function is defined as $$Q : S \times A \rightarrow \mathbb{R}$$ $$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma \max\limits_{a} Q(s_{t+1},a) - Q(s_t,a_t)] $$ $\alpha$ and $\gamma$...
satya's user avatar
  • 187
0 votes
1 answer
267 views

How would you shape a reward function if there was four quantities to optimize?

I found this article quite useful on how to shape a reward function in RL. However, the example they gave is quite simple, where the goal is to minimize only two quantities (velocity and distance). ...
BAKYAC's user avatar
  • 36
0 votes
1 answer
221 views

How do we get the value of this state of an MDP, at time-step $h-2$, using dynamic programming?

I am trying to understand the problem below, represented as an MDP with four states (PU, PF, RU, and RF) and two actions (AS). Let's consider V(RF), the value of the state RF. At time-step $h$, V(RF) ...
AHF's user avatar
  • 61
2 votes
1 answer
670 views

What happens with policy gradient methods if rewards are differentiable?

I would like some help with understanding why there is no explicit flow of information from the reward gradient to the parameters of the policy in policy gradient methods. What I mean is the following,...
External Supplier Staff's user avatar
1 vote
1 answer
597 views

How to scale all positive continuous reward?

My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between ...
fardis nadimi's user avatar
3 votes
1 answer
312 views

Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward

In the appendix of the Constrained Policy Optimization (CPO) paper (Arxiv), the authors denote the discounted future state distribution $d^\pi$ as: $$d^\pi(s) = (1-\gamma) \sum_{t=0}^\infty{\gamma^t P(...
josealeixo.pc's user avatar
6 votes
1 answer
1k views

How to improve the reward signal when the rewards are sparse?

In cases where the reward is delayed, this can negatively impact a models ability to do proper credit assignment. In the case of a sparse reward, are there ways in which this can be negated? In a ...
tryingtolearn's user avatar
2 votes
1 answer
188 views

Can the rewards be matrices when using DQN?

I have a basic question. I'm working towards developing a reward function for my DQN. I'd like to train an RL agent to edit pixels on an image. I understand that convolutions are ideal for working ...
junfanbl's user avatar
  • 323
2 votes
1 answer
133 views

How can I go from $R(s)$ to $R(s,a)$ in this specific MDP?

I'm trying to implement a research paper, as explained in this other post, here the author of the paper assumed R as a function of both states and actions, while the code (and the MDP) I'm using to ...
ИванКарамазов's user avatar
3 votes
0 answers
137 views

Is better to reward short- or long-term progress in Q-learning?

I have been training some kind of agent to reach a target using a Q-learning based approach, and I have tried two different types of rewards: Long-term reward: $\mathrm{reward} = - \mathrm{distance}(\...
Thomas Wagenaar's user avatar
0 votes
0 answers
213 views

Is my reward function non-Markovian?

I am working on an RL problem where the time when the agent obtains the reward for taking action $a$ in time step $t$ is stochastic. In fact, there is no immediate reward for taking action $a$ in time ...
shirin elahi's user avatar
1 vote
1 answer
212 views

If the reward function of an environment depends on some initial conditions, should I create a separate environment for each condition?

I would like some guidance on how to design an Environment for a Reinforcement Learning agent where the stopping conditions and rewards for the environment change based on an initial set of input ...
RL_NOOB's user avatar
  • 15