Questions tagged [rewards]

For questions related to the concept of reward, for example, in the context of reinforcement learning and Markov decision processes. For questions related to reward functions, reward design, reward shaping, reward hacking, etc., there are also those specific tags, so you use them instead of this generic one, unless your question is also about the concept of reward.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

in the "reward to go" trick in policy gradient methods, I have a question about the proof?

I am specifically talking about this proof Why does the "reward to go" trick in policy gradient methods work? where Dennis says 'on ith iteration the outer sum of random variable and ...
abhilash sharma's user avatar
1 vote
0 answers
49 views

Reward shaping for dense and sparse rewards

I am working on an RL Problem that drives me nuts. My goal is to control a robot arm in a simulator that has to do 2 things: Hold the arm in a certain position (that is easy and done) If I apply an ...
mavex857's user avatar
3 votes
3 answers
473 views

What are reward networks in reinforcement learning?

I am reading the following article given over here - The goal of both inverse reinforcement learning (IRL) algorithms (e.g. AIRL, GAIL) and preference comparison is to discover a reward function. In ...
desert_ranger's user avatar
3 votes
1 answer
108 views

How does changing the order of a sequence of rewards affect the cumulative discounted reward

Let's say from time step $t$, the agent interact with the environment for $n$ steps, and gets a reward seuqence $R = (r_{t}, r_{t+1}, \cdots , r_{t+n})$ Now we respectively sort the reward sequence $R$...
juicyliberty's user avatar
0 votes
0 answers
25 views

Using multi-dimensional rewards to prevent intermediary reward bias

I came up with a new type of reward scheme to avoid intermediary reward bias and am unable to find any literature references to it. My questions are Is this novel? Is this useful? The approach is as ...
Maarten's user avatar
2 votes
2 answers
242 views

One to one relation between state + action -> reward

I am designing my own environment for a specific problem and I am thinking of the reward function for it. In some RL algos it is common to learn the reward that is associated with taking an action ...
Erik Storm's user avatar
2 votes
1 answer
298 views

What's the architecture and size of neural-network-based reward models as used in reinforcement learning by human feedback

My rough understanding of RLHF as used for ChatGPT in a nutshell is this: A reward model is trained using comparisons of different responses to the same prompt. Human trainers rank these responses ...
Hans-Peter Stricker's user avatar
3 votes
0 answers
241 views

Let's Verify Step by Step: Old wine in new bottles?

In their paper "Let's Verify Step by Step" OpenAI proudly presents a new way of reward learning which shall foster LLMs' capabilities of mathematical and logical reasoning: We've trained a ...
Hans-Peter Stricker's user avatar
0 votes
0 answers
37 views

How to evaluate the performance of off-line & model-free reinforcement leaning?

I'm currently studying on off-line reinforcement learning (RL) and trying to utilize it for medical data. Because it seemed hard to develop well-performing environment model, I decided to adopt model-...
Maverick's user avatar
1 vote
0 answers
281 views

MDP with a non-markovian reward function?

I have set up a RL environment and it converges to a decent solution when using a reward function: $R(s_t,a_t) = fenv(s_t, a_t)$ , where $fenv$ is the environment dynamics. Now, i want to change the ...
StarDust_08's user avatar
1 vote
1 answer
256 views

How to normalizing various elements of the reward function?

Suppose I have a reward function $R$ that I wish to penalize w.r.t two distinct phenomenons $A$ and $B$. $A$, for example, could represent the phenomenon of the state not crossing some boundary $[s_1,...
Hadar Sharvit's user avatar
0 votes
0 answers
69 views

How do I improve the reward of policy gradient network when multiple states and actions exist per time step?

I am working on a project, in which I'm using a policy gradient algorithm (REINFORCE) to select the best cleaning method/methods for erroneous samples in tabular datasets. The details are as follows. ...
aby's user avatar
  • 1
0 votes
1 answer
180 views

Should the concept of discounted rewards result in multiple arrays per episode in RL?

Note that I'm coming from mostly only working with the REINFORCE algorithm, but I've typically seen discounted rewards calculated in a way that looks like below: Say you have a reward array of length <...
Josh's user avatar
  • 89
0 votes
0 answers
143 views

Shaping reward so that it maximizes multiple components together

I am fairly new to RL and I compete in AWS DeepRacer student league. The main task there is to create a reward function. All the hyperparameters and action space are fixed. So far, I know how to shape ...
aniketvp24's user avatar
2 votes
1 answer
933 views

How to deal with small reward values

In my environment rewards are generally small, e.g. [-0.01, 0.01]. My concern is that small reward values might get dominated or distorted by the noise during the training. Does it make sense to scale ...
Mika's user avatar
  • 341
0 votes
0 answers
35 views

Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ in terms of $p$ (3.4) and $r$ (3.5) [duplicate]

I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise: Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ ...
user's user avatar
  • 145
4 votes
2 answers
216 views

$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?

I would like to solve the first question of Exercise 3.19 from Sutton and Barto: Exercise 3.19 The value of an action, $q_{\pi}(s, a)$, depends on the expected next reward and the expected sum of the ...
user's user avatar
  • 145
1 vote
1 answer
297 views

Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?

The creation of negative rewards leads to the chance of Q-values being negative. However, networks with relu or sigmoid activations, just cannot predict negative values. This will lead to a case where ...
desert_ranger's user avatar
0 votes
0 answers
240 views

How to solve a reinforcement learning problem with a stochastic reward function?

In a discrete time system, an environment has an unknown reward probability $p(r|s,a)$. However, the transition probability $p(s'\mid s,a)$ is deterministic. In my case, the reward for the same action ...
Zhenzhen Gong's user avatar
2 votes
0 answers
270 views

How to solve a reinforcement learning problem with changing rewards?

I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for ...
Zhenzhen Gong's user avatar
0 votes
0 answers
1k views

PPO: how to scale rewards

I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried ...
Antonis Karvelas's user avatar
3 votes
1 answer
362 views

Why does the average-reward estimator for continuing tasks use the TD error?

In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q_\pi(s,a)$ and value function $v_\pi(s)$ ...
kmf's user avatar
  • 106
2 votes
1 answer
713 views

Why is training longer not better in reinforcement learning?

I have trained an RL agent (PPO) for 6 million steps to solve the OpenAI gym LunarLander-v2. Surprisingly, the agent performs best already after 320K steps and is getting worse after that. In the ...
Martin S's user avatar
  • 213
1 vote
2 answers
1k views

What is the difference between a policy and rewards?

I don't understand the difference between a policy and rewards. Sure, a policy tells us what to do, but isn't the output of a neural network trained on rewards basically a policy (i.e. choose the ...
Antonis Karvelas's user avatar
1 vote
0 answers
782 views

What is commonly done for standardization/normalization of the targets in Deep Q-Learning?

I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
Peter's user avatar
  • 55
1 vote
0 answers
111 views

Normalisation of reward function

Problem Currently, I have some problems defining a reward function for my RL project and mainly with how to normalise the score such that the highest possible score for all instances of the ...
Jesse's user avatar
  • 31
7 votes
2 answers
8k views

What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?

In Deep Reinforcement Learning (DRL) I am having difficulties in understanding the difference between a Loss function, a reward/penalty and the integration of both in DRL. Loss function: Given an ...
Theo Deep's user avatar
  • 185
0 votes
0 answers
210 views

How to pass the rewards in zero-sum multiplayer context when using REINFORCE?

Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm. I have doubts about passing reward values in ...
hanugm's user avatar
  • 3,840
1 vote
0 answers
59 views

Generative systems based on Schmidhuber's compression framework

In Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes ...
2080's user avatar
  • 121
0 votes
1 answer
1k views

Is it a bad practice to use cumulative rewards in reinforcement learning

I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over ...
Leibniz's user avatar
  • 69
1 vote
1 answer
409 views

How are rewards calculated for episodic tasks like playing chess or tic-tac-toe?

I am new to Reinforcement Learning and trying to understand the concept of reaping rewards during episodic tasks. I think in games like tic-tac-toe, rewards will be in terms of a win or lose. But does ...
Saily_Shah's user avatar
1 vote
1 answer
88 views

In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?

I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the ...
Hermes Morales's user avatar
0 votes
0 answers
534 views

How to normalize rewards in REINFORCE?

I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to ...
Mastiff's user avatar
  • 121
3 votes
1 answer
228 views

How do I represent sample efficiency of RL rewards in mathematical notation?

I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
bluewander's user avatar
1 vote
2 answers
1k views

What is the difference between a reward and a value for a given state?

I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I ...
dcr's user avatar
  • 57
6 votes
1 answer
153 views

Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?

Say I've got two Markov Decision Processes (MDPs): $$\mathcal{M_0} = (\mathcal{S}, \mathcal{A}, P, R_0),\quad\text{and}\quad\mathcal{M}_1 = (\mathcal{S}, \mathcal{A}, P, R_1)$$ Both have the same set ...
Kostya's user avatar
  • 2,524
2 votes
1 answer
563 views

Difference in UCB performance when scaling the rewards

I notice the following behavior when running experiments with $\epsilon$-greedy and UCB1. If the reward is kept binary (0 or 1) both algorithm's performances are on par with each other. However, if I ...
d56's user avatar
  • 233
0 votes
1 answer
163 views

Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning

I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0. However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) ...
SJa's user avatar
  • 393
2 votes
0 answers
69 views

What is the dimensionality of these derivatives in the paper "Active Learning for Reward Estimation in Inverse Reinforcement Learning"?

I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning. I'm specifically referring to section 2.3 of the paper. Let's ...
ИванКарамазов's user avatar
2 votes
1 answer
102 views

Is there multi-agent reinforcement learning model in which (some of the) reward is given by other agent and not by the external environment?

The traditional setting of multiagent reinforcement learning (MARL) is the mode in which there is set of agents and external environment. And the reward is given to each agent - individually or ...
TomR's user avatar
  • 843
3 votes
1 answer
154 views

How do we derive the expression for average reward setting in continuing tasks?

In the average reward setting we have: $$r(\pi)\doteq \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{t=1}^{h}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$ $$r(\pi)\doteq \lim_{t\rightarrow\infty}\mathbb{E}[R_{t}...
ZERO NULLS's user avatar
3 votes
1 answer
3k views

Why do my rewards reduce after extensive training using D3QN?

I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually ...
desert_ranger's user avatar
4 votes
2 answers
756 views

Why is regret so defined in MABs?

Consider a multi-armed bandit(MAB). There are $k$ arms, with reward distributions $R_i$ where $1 \leq i \leq k$. Let $\mu_i$ denote the mean of the $i^{th}$ distribution. If we run the multi-armed ...
stoic-santiago's user avatar
1 vote
0 answers
207 views

Why would DDPG with Hindsight Experience Replay not converge?

I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
Vedant Shah's user avatar
3 votes
1 answer
268 views

How can I fix jerky movement in a continuous action space

I am training an agent to do object avoidance. The agent has control over its steering angle and its speed. The steering angle and speed are normalized in a $[−1,1]$ range, where the sign encodes ...
Shon Verch's user avatar
3 votes
1 answer
387 views

How do I design the rewards and penalties for an agent whose goal it is to explore a map

I am trying to train an agent to explore an unknown two-dimensional map while avoiding circular obstacles (with varying radii). The agent has control over its steering angle and its speed. The ...
Shon Verch's user avatar
2 votes
0 answers
129 views

Should I use the discounted average reward as objective in a finite-horizon problem?

I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
lll's user avatar
  • 121
5 votes
2 answers
192 views

How can we prevent AGI from doing drugs?

I recently read some introductions to AI alignment, AIXI and decision theory things. As far as I understood, one of the main problems in AI alignment is how to define a utility function well, not ...
user3584499's user avatar
13 votes
3 answers
2k views

Why is the reward in reinforcement learning always a scalar?

I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
Sid Mani's user avatar
  • 233
4 votes
3 answers
1k views

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem?

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem? For example, you want to train a DQN agent in an environment, and you want to know what the highest ...
user avatar