Questions tagged [rewards]
For questions related to the concept of reward, for example, in the context of reinforcement learning and Markov decision processes. For questions related to reward functions, reward design, reward shaping, reward hacking, etc., there are also those specific tags, so you use them instead of this generic one, unless your question is also about the concept of reward.
121
questions
1
vote
0
answers
36
views
Reward shaping for dense and sparse rewards
I am working on an RL Problem that drives me nuts.
My goal is to control a robot arm in a simulator that has to do 2 things:
Hold the arm in a certain position (that is easy and done)
If I apply an ...
3
votes
3
answers
460
views
What are reward networks in reinforcement learning?
I am reading the following article given over here -
The goal of both inverse reinforcement learning (IRL) algorithms (e.g.
AIRL, GAIL) and preference comparison is to discover a reward
function. In ...
3
votes
1
answer
103
views
How does changing the order of a sequence of rewards affect the cumulative discounted reward
Let's say from time step $t$, the agent interact with the environment for $n$ steps, and gets a reward seuqence $R = (r_{t}, r_{t+1}, \cdots , r_{t+n})$
Now we respectively sort the reward sequence $R$...
0
votes
0
answers
24
views
Using multi-dimensional rewards to prevent intermediary reward bias
I came up with a new type of reward scheme to avoid intermediary reward bias and am unable to find any literature references to it. My questions are
Is this novel?
Is this useful?
The approach is as ...
2
votes
2
answers
233
views
One to one relation between state + action -> reward
I am designing my own environment for a specific problem and I am thinking of the reward function for it. In some RL algos it is common to learn the reward that is associated with taking an action ...
2
votes
1
answer
268
views
What's the architecture and size of neural-network-based reward models as used in reinforcement learning by human feedback
My rough understanding of RLHF as used for ChatGPT in a nutshell is this:
A reward model is trained using comparisons of different responses
to the same prompt. Human trainers rank these responses ...
3
votes
0
answers
236
views
Let's Verify Step by Step: Old wine in new bottles?
In their paper "Let's Verify Step by Step" OpenAI proudly presents a new way of reward learning which shall foster LLMs' capabilities of mathematical and logical reasoning:
We've trained a ...
0
votes
0
answers
36
views
How to evaluate the performance of off-line & model-free reinforcement leaning?
I'm currently studying on off-line reinforcement learning (RL) and trying to utilize it for medical data. Because it seemed hard to develop well-performing environment model, I decided to adopt model-...
1
vote
0
answers
270
views
MDP with a non-markovian reward function?
I have set up a RL environment and it converges to a decent solution when using a reward function:
$R(s_t,a_t) = fenv(s_t, a_t)$ , where $fenv$ is the environment dynamics.
Now, i want to change the ...
1
vote
1
answer
230
views
How to normalizing various elements of the reward function?
Suppose I have a reward function $R$ that I wish to penalize w.r.t two distinct phenomenons $A$ and $B$. $A$, for example, could represent the phenomenon of the state not crossing some boundary $[s_1,...
0
votes
0
answers
67
views
How do I improve the reward of policy gradient network when multiple states and actions exist per time step?
I am working on a project, in which I'm using a policy gradient algorithm (REINFORCE) to select the best cleaning method/methods for erroneous samples in tabular datasets.
The details are as follows.
...
0
votes
1
answer
176
views
Should the concept of discounted rewards result in multiple arrays per episode in RL?
Note that I'm coming from mostly only working with the REINFORCE algorithm, but I've typically seen discounted rewards calculated in a way that looks like below:
Say you have a reward array of length <...
0
votes
0
answers
138
views
Shaping reward so that it maximizes multiple components together
I am fairly new to RL and I compete in AWS DeepRacer student league. The main task there is to create a reward function. All the hyperparameters and action space are fixed. So far, I know how to shape ...
2
votes
1
answer
879
views
How to deal with small reward values
In my environment rewards are generally small, e.g. [-0.01, 0.01]. My concern is that small reward values might get dominated or distorted by the noise during the training. Does it make sense to scale ...
0
votes
0
answers
35
views
Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ in terms of $p$ (3.4) and $r$ (3.5) [duplicate]
I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:
Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ ...
4
votes
2
answers
212
views
$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?
I would like to solve the first question of Exercise 3.19 from Sutton and Barto:
Exercise 3.19 The value of an action, $q_{\pi}(s, a)$, depends on the expected next reward and
the expected sum of the ...
1
vote
1
answer
290
views
Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?
The creation of negative rewards leads to the chance of Q-values being negative. However, networks with relu or sigmoid activations, just cannot predict negative values. This will lead to a case where ...
0
votes
0
answers
236
views
How to solve a reinforcement learning problem with a stochastic reward function?
In a discrete time system, an environment has an unknown reward probability $p(r|s,a)$. However, the transition probability $p(s'\mid s,a)$ is deterministic.
In my case, the reward for the same action ...
2
votes
0
answers
267
views
How to solve a reinforcement learning problem with changing rewards?
I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for ...
0
votes
0
answers
1k
views
PPO: how to scale rewards
I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried ...
3
votes
1
answer
355
views
Why does the average-reward estimator for continuing tasks use the TD error?
In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q_\pi(s,a)$ and value function $v_\pi(s)$ ...
2
votes
1
answer
677
views
Why is training longer not better in reinforcement learning?
I have trained an RL agent (PPO) for 6 million steps to solve the OpenAI gym LunarLander-v2. Surprisingly, the agent performs best already after 320K steps and is getting worse after that.
In the ...
1
vote
2
answers
1k
views
What is the difference between a policy and rewards?
I don't understand the difference between a policy and rewards. Sure, a policy tells us what to do, but isn't the output of a neural network trained on rewards basically a policy (i.e. choose the ...
1
vote
0
answers
752
views
What is commonly done for standardization/normalization of the targets in Deep Q-Learning?
I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
1
vote
0
answers
107
views
Normalisation of reward function
Problem
Currently, I have some problems defining a reward function for my RL project and mainly with how to normalise the score such that the highest possible score for all instances of the ...
7
votes
2
answers
8k
views
What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?
In Deep Reinforcement Learning (DRL) I am having difficulties in understanding the difference between a Loss function, a reward/penalty and the integration of both in DRL.
Loss function: Given an ...
0
votes
0
answers
207
views
How to pass the rewards in zero-sum multiplayer context when using REINFORCE?
Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm.
I have doubts about passing reward values in ...
1
vote
0
answers
59
views
Generative systems based on Schmidhuber's compression framework
In Driven by Compression Progress:
A Simple Principle Explains Essential Aspects of
Subjective Beauty, Novelty, Surprise,
Interestingness, Attention, Curiosity, Creativity,
Art, Science, Music, Jokes ...
0
votes
1
answer
1k
views
Is it a bad practice to use cumulative rewards in reinforcement learning
I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over ...
1
vote
1
answer
397
views
How are rewards calculated for episodic tasks like playing chess or tic-tac-toe?
I am new to Reinforcement Learning and trying to understand the concept of reaping rewards during episodic tasks. I think in games like tic-tac-toe, rewards will be in terms of a win or lose. But does ...
1
vote
1
answer
86
views
In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?
I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the ...
0
votes
0
answers
526
views
How to normalize rewards in REINFORCE?
I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to ...
3
votes
1
answer
228
views
How do I represent sample efficiency of RL rewards in mathematical notation?
I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
1
vote
2
answers
1k
views
What is the difference between a reward and a value for a given state?
I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I ...
6
votes
1
answer
151
views
Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?
Say I've got two Markov Decision Processes (MDPs):
$$\mathcal{M_0} = (\mathcal{S}, \mathcal{A}, P, R_0),\quad\text{and}\quad\mathcal{M}_1 = (\mathcal{S}, \mathcal{A}, P, R_1)$$
Both have the same set ...
2
votes
1
answer
543
views
Difference in UCB performance when scaling the rewards
I notice the following behavior when running experiments with $\epsilon$-greedy and UCB1. If the reward is kept binary (0 or 1) both algorithm's performances are on par with each other. However, if I ...
0
votes
1
answer
162
views
Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning
I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0.
However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) ...
2
votes
0
answers
69
views
What is the dimensionality of these derivatives in the paper "Active Learning for Reward Estimation in Inverse Reinforcement Learning"?
I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning.
I'm specifically referring to section 2.3 of the paper.
Let's ...
2
votes
1
answer
102
views
Is there multi-agent reinforcement learning model in which (some of the) reward is given by other agent and not by the external environment?
The traditional setting of multiagent reinforcement learning (MARL) is the mode in which there is set of agents and external environment. And the reward is given to each agent - individually or ...
3
votes
1
answer
154
views
How do we derive the expression for average reward setting in continuing tasks?
In the average reward setting we have:
$$r(\pi)\doteq \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{t=1}^{h}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$
$$r(\pi)\doteq \lim_{t\rightarrow\infty}\mathbb{E}[R_{t}...
3
votes
1
answer
3k
views
Why do my rewards reduce after extensive training using D3QN?
I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually ...
4
votes
2
answers
731
views
Why is regret so defined in MABs?
Consider a multi-armed bandit(MAB). There are $k$ arms, with reward distributions $R_i$ where $1 \leq i \leq k$. Let $\mu_i$ denote the mean of the $i^{th}$ distribution.
If we run the multi-armed ...
1
vote
0
answers
200
views
Why would DDPG with Hindsight Experience Replay not converge?
I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
3
votes
1
answer
267
views
How can I fix jerky movement in a continuous action space
I am training an agent to do object avoidance. The agent has control over its steering angle and its speed. The steering angle and speed are normalized in a $[−1,1]$ range, where the sign encodes ...
3
votes
1
answer
367
views
How do I design the rewards and penalties for an agent whose goal it is to explore a map
I am trying to train an agent to explore an unknown two-dimensional map while avoiding circular obstacles (with varying radii). The agent has control over its steering angle and its speed. The ...
2
votes
0
answers
129
views
Should I use the discounted average reward as objective in a finite-horizon problem?
I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
5
votes
2
answers
192
views
How can we prevent AGI from doing drugs?
I recently read some introductions to AI alignment, AIXI and decision theory things.
As far as I understood, one of the main problems in AI alignment is how to define a utility function well, not ...
13
votes
3
answers
2k
views
Why is the reward in reinforcement learning always a scalar?
I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
4
votes
3
answers
1k
views
Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem?
Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem?
For example, you want to train a DQN agent in an environment, and you want to know what the highest ...
2
votes
1
answer
3k
views
Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking ...