As of May 31, 2023, we have updated our Code of Conduct.

Questions tagged [rewards]

For questions related to the concept of reward, for example, in the context of reinforcement learning and Markov decision processes. For questions related to reward functions, reward design, reward shaping, reward hacking, etc., there are also those specific tags, so you use them instead of this generic one, unless your question is also about the concept of reward.

Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

Penalizing invalid action and ending episode early causes agent to end episode with invalid action in a non-goal state?

Domain: start at initial position, navigate to goal-position with N,S,E,W actions. Algorithm: PPO Libs: custom gym-env, stable baselines3 Penalties: valid step: -1 (promote shortest path) INvalid step:...
marshacker's user avatar
0 votes
0 answers
31 views

How to evaluate the performance of off-line & model-free reinforcement leaning?

I'm currently studying on off-line reinforcement learning (RL) and trying to utilize it for medical data. Because it seemed hard to develop well-performing environment model, I decided to adopt model-...
Maverick's user avatar
1 vote
0 answers
145 views

MDP with a non-markovian reward function?

I have set up a RL environment and it converges to a decent solution when using a reward function: $R(s_t,a_t) = fenv(s_t, a_t)$ , where $fenv$ is the environment dynamics. Now, i want to change the ...
StarDust_08's user avatar
1 vote
1 answer
94 views

How to normalizing various elements of the reward function?

Suppose I have a reward function $R$ that I wish to penalize w.r.t two distinct phenomenons $A$ and $B$. $A$, for example, could represent the phenomenon of the state not crossing some boundary $[s_1,...
Hadar Sharvit's user avatar
0 votes
0 answers
41 views

Frozen baseline for policy gradient rewards

I have a continuous reinforcement learning problem for which I use policy gradients and I use a baseline to decrease the variance of the gradients. The baseline that I used is the moving average of ...
aby's user avatar
  • 1
0 votes
0 answers
55 views

How do I improve the reward of policy gradient network when multiple states and actions exist per time step?

I am working on a project, in which I'm using a policy gradient algorithm (REINFORCE) to select the best cleaning method/methods for erroneous samples in tabular datasets. The details are as follows. ...
aby's user avatar
  • 1
0 votes
1 answer
114 views

Should the concept of discounted rewards result in multiple arrays per episode in RL?

Note that I'm coming from mostly only working with the REINFORCE algorithm, but I've typically seen discounted rewards calculated in a way that looks like below: Say you have a reward array of length <...
Josh's user avatar
  • 89
0 votes
0 answers
82 views

Shaping reward so that it maximizes multiple components together

I am fairly new to RL and I compete in AWS DeepRacer student league. The main task there is to create a reward function. All the hyperparameters and action space are fixed. So far, I know how to shape ...
aniketvp24's user avatar
0 votes
0 answers
242 views

Reward shaping for an autonomous driving car (AWS DeepRacer)

From past 2 months, I am competing in AWS Deepracer student league. I was new to RL but had some knowledge in supervised and unsupervised learning. In the league the hyperparameters and action space ...
aniketvp24's user avatar
0 votes
0 answers
61 views

When use positive or negative rewards in reinforcement learning? Is there anything in literature?

Let's say I can design a reward as function of a distance $d>0$ from the target in 2 ways: $r=\frac{1}{1+d}$ or $r=-d$. The first is defined in $(0,1]$ the second in $(-\infty,0]$. I would expect ...
pippo's user avatar
  • 41
0 votes
0 answers
27 views

How to design a DQN that needs to self-play with very delayed rewards?

I got a single-player card game (comparable to BlackJack) in which rewards are VERY delayed. On average a player can estimate the effect of his move fifteen steps deep. As there are 52 cards to play ...
J. M. Arnold's user avatar
0 votes
0 answers
27 views

Q-learning: In tic-tac-toe, how to choose rewards values?

| I'm experimenting for first times in neural networks and Q-learning. In tic-tac-toe game, each move has 5 possible esits invalid move valid move, match is still open draft you lose you win ...
realtebo's user avatar
  • 101
2 votes
1 answer
532 views

How to deal with small reward values

In my environment rewards are generally small, e.g. [-0.01, 0.01]. My concern is that small reward values might get dominated or distorted by the noise during the training. Does it make sense to scale ...
Mika's user avatar
  • 331
0 votes
0 answers
29 views

Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ in terms of $p$ (3.4) and $r$ (3.5) [duplicate]

I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise: Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ ...
user's user avatar
  • 95
4 votes
1 answer
101 views

$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?

I would like to solve the first question of Exercise 3.19 from Sutton and Barto: Exercise 3.19 The value of an action, $q_{\pi}(s, a)$, depends on the expected next reward and the expected sum of the ...
user's user avatar
  • 95
1 vote
1 answer
165 views

Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?

The creation of negative rewards leads to the chance of Q-values being negative. However, networks with relu or sigmoid activations, just cannot predict negative values. This will lead to a case where ...
desert_ranger's user avatar
0 votes
0 answers
180 views

How to solve a reinforcement learning problem with a stochastic reward function?

In a discrete time system, an environment has an unknown reward probability $p(r|s,a)$. However, the transition probability $p(s'\mid s,a)$ is deterministic. In my case, the reward for the same action ...
Zhenzhen Gong's user avatar
2 votes
0 answers
208 views

How to solve a reinforcement learning problem with changing rewards?

I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for ...
Zhenzhen Gong's user avatar
0 votes
0 answers
770 views

PPO: how to scale rewards

I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried ...
Antonis Karvelas's user avatar
2 votes
1 answer
255 views

Why does the average-reward estimator for continuing tasks use the TD error?

In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q_\pi(s,a)$ and value function $v_\pi(s)$ ...
kmf's user avatar
  • 86
2 votes
1 answer
473 views

Why is training longer not better in reinforcement learning?

I have trained an RL agent (PPO) for 6 million steps to solve the OpenAI gym LunarLander-v2. Surprisingly, the agent performs best already after 320K steps and is getting worse after that. In the ...
Martin S's user avatar
  • 213
1 vote
2 answers
610 views

What is the difference between a policy and rewards?

I don't understand the difference between a policy and rewards. Sure, a policy tells us what to do, but isn't the output of a neural network trained on rewards basically a policy (i.e. choose the ...
Antonis Karvelas's user avatar
1 vote
0 answers
455 views

What is commonly done for standardization/normalization of the targets in Deep Q-Learning?

I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
Peter's user avatar
  • 55
1 vote
0 answers
64 views

Normalisation of reward function

Problem Currently, I have some problems defining a reward function for my RL project and mainly with how to normalise the score such that the highest possible score for all instances of the ...
Jesse's user avatar
  • 31
5 votes
2 answers
4k views

What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?

In Deep Reinforcement Learning (DRL) I am having difficulties in understanding the difference between a Loss function, a reward/penalty and the integration of both in DRL. Loss function: Given an ...
Theo Deep's user avatar
  • 165
0 votes
0 answers
155 views

How to pass the rewards in zero-sum multiplayer context when using REINFORCE?

Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm. I have doubts about passing reward values in ...
hanugm's user avatar
  • 3,531
1 vote
0 answers
49 views

Generative systems based on Schmidhuber's compression framework

In Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes ...
2080's user avatar
  • 121
0 votes
1 answer
616 views

Is it a bad practice to use cumulative rewards in reinforcement learning

I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over ...
Leibniz's user avatar
  • 69
1 vote
1 answer
276 views

How are rewards calculated for episodic tasks like playing chess or tic-tac-toe?

I am new to Reinforcement Learning and trying to understand the concept of reaping rewards during episodic tasks. I think in games like tic-tac-toe, rewards will be in terms of a win or lose. But does ...
Saily_Shah's user avatar
1 vote
1 answer
69 views

In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?

I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the ...
Hermes Morales's user avatar
0 votes
0 answers
370 views

How to normalize rewards in REINFORCE?

I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to ...
Mastiff's user avatar
  • 121
3 votes
1 answer
203 views

How do I represent sample efficiency of RL rewards in mathematical notation?

I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
bluewander's user avatar
1 vote
2 answers
1k views

What is the difference between a reward and a value for a given state?

I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I ...
dcr's user avatar
  • 57
6 votes
1 answer
122 views

Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?

Say I've got two Markov Decision Processes (MDPs): $$\mathcal{M_0} = (\mathcal{S}, \mathcal{A}, P, R_0),\quad\text{and}\quad\mathcal{M}_1 = (\mathcal{S}, \mathcal{A}, P, R_1)$$ Both have the same set ...
Kostya's user avatar
  • 2,380
2 votes
1 answer
349 views

Difference in UCB performance when scaling the rewards

I notice the following behavior when running experiments with $\epsilon$-greedy and UCB1. If the reward is kept binary (0 or 1) both algorithm's performances are on par with each other. However, if I ...
d56's user avatar
  • 223
0 votes
1 answer
134 views

Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning

I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0. However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) ...
SJa's user avatar
  • 371
2 votes
0 answers
67 views

What is the dimensionality of these derivatives in the paper "Active Learning for Reward Estimation in Inverse Reinforcement Learning"?

I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning. I'm specifically referring to section 2.3 of the paper. Let's ...
ИванКарамазов's user avatar
2 votes
1 answer
87 views

Is there multi-agent reinforcement learning model in which (some of the) reward is given by other agent and not by the external environment?

The traditional setting of multiagent reinforcement learning (MARL) is the mode in which there is set of agents and external environment. And the reward is given to each agent - individually or ...
TomR's user avatar
  • 813
3 votes
1 answer
137 views

How do we derive the expression for average reward setting in continuing tasks?

In the average reward setting we have: $$r(\pi)\doteq \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{t=1}^{h}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$ $$r(\pi)\doteq \lim_{t\rightarrow\infty}\mathbb{E}[R_{t}...
ZERO NULLS's user avatar
3 votes
1 answer
2k views

Why do my rewards reduce after extensive training using D3QN?

I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually ...
desert_ranger's user avatar
4 votes
2 answers
530 views

Why is regret so defined in MABs?

Consider a multi-armed bandit(MAB). There are $k$ arms, with reward distributions $R_i$ where $1 \leq i \leq k$. Let $\mu_i$ denote the mean of the $i^{th}$ distribution. If we run the multi-armed ...
esoteric-elliptic's user avatar
1 vote
0 answers
162 views

Why would DDPG with Hindsight Experience Replay not converge?

I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
Vedant Shah's user avatar
3 votes
1 answer
209 views

How can I fix jerky movement in a continuous action space

I am training an agent to do object avoidance. The agent has control over its steering angle and its speed. The steering angle and speed are normalized in a $[−1,1]$ range, where the sign encodes ...
Shon Verch's user avatar
3 votes
1 answer
210 views

How do I design the rewards and penalties for an agent whose goal it is to explore a map

I am trying to train an agent to explore an unknown two-dimensional map while avoiding circular obstacles (with varying radii). The agent has control over its steering angle and its speed. The ...
Shon Verch's user avatar
2 votes
0 answers
112 views

Should I use the discounted average reward as objective in a finite-horizon problem?

I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
lll's user avatar
  • 121
4 votes
2 answers
176 views

How can we prevent AGI from doing drugs?

I recently read some introductions to AI alignment, AIXI and decision theory things. As far as I understood, one of the main problems in AI alignment is how to define a utility function well, not ...
user3584499's user avatar
12 votes
3 answers
1k views

Why is the reward in reinforcement learning always a scalar?

I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
Sid Mani's user avatar
  • 223
4 votes
3 answers
965 views

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem?

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem? For example, you want to train a DQN agent in an environment, and you want to know what the highest ...
user avatar
2 votes
1 answer
2k views

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking ...
THAT_AI_GUY's user avatar
3 votes
1 answer
650 views

How is the reward in reinforcement learning different from the label in supervised learning problems?

How is the notion of immediate reward used in the reinforcement learning different from the notion of a label we find in the supervised learning problems?
Saptam's user avatar
  • 67