Questions tagged [rewards]
For questions related to the concept of reward, for example, in the context of reinforcement learning and Markov decision processes. For questions related to reward functions, reward design, reward shaping, reward hacking, etc., there are also those specific tags, so you use them instead of this generic one, unless your question is also about the concept of reward.
118
questions
2
votes
2
answers
216
views
One to one relation between state + action -> reward
I am designing my own environment for a specific problem and I am thinking of the reward function for it. In some RL algos it is common to learn the reward that is associated with taking an action ...
- 61
2
votes
1
answer
170
views
What's the architecture and size of neural-network-based reward models as used in reinforcement learning by human feedback
My rough understanding of RLHF as used for ChatGPT in a nutshell is this:
A reward model is trained using comparisons of different responses
to the same prompt. Human trainers rank these responses ...
3
votes
0
answers
187
views
Let's Verify Step by Step: Old wine in new bottles?
In their paper "Let's Verify Step by Step" OpenAI proudly presents a new way of reward learning which shall foster LLMs' capabilities of mathematical and logical reasoning:
We've trained a ...
0
votes
0
answers
45
views
Penalizing invalid action and ending episode early causes agent to end episode with invalid action in a non-goal state?
Domain: start at initial position, navigate to goal-position with N,S,E,W actions.
Algorithm: PPO
Libs: custom gym-env, stable baselines3
Penalties:
valid step: -1 (promote shortest path)
INvalid step:...
0
votes
0
answers
34
views
How to evaluate the performance of off-line & model-free reinforcement leaning?
I'm currently studying on off-line reinforcement learning (RL) and trying to utilize it for medical data. Because it seemed hard to develop well-performing environment model, I decided to adopt model-...
- 1
1
vote
0
answers
226
views
MDP with a non-markovian reward function?
I have set up a RL environment and it converges to a decent solution when using a reward function:
$R(s_t,a_t) = fenv(s_t, a_t)$ , where $fenv$ is the environment dynamics.
Now, i want to change the ...
- 11
1
vote
1
answer
179
views
How to normalizing various elements of the reward function?
Suppose I have a reward function $R$ that I wish to penalize w.r.t two distinct phenomenons $A$ and $B$. $A$, for example, could represent the phenomenon of the state not crossing some boundary $[s_1,...
- 371
0
votes
0
answers
64
views
How do I improve the reward of policy gradient network when multiple states and actions exist per time step?
I am working on a project, in which I'm using a policy gradient algorithm (REINFORCE) to select the best cleaning method/methods for erroneous samples in tabular datasets.
The details are as follows.
...
- 1
0
votes
1
answer
139
views
Should the concept of discounted rewards result in multiple arrays per episode in RL?
Note that I'm coming from mostly only working with the REINFORCE algorithm, but I've typically seen discounted rewards calculated in a way that looks like below:
Say you have a reward array of length <...
- 89
0
votes
0
answers
124
views
Shaping reward so that it maximizes multiple components together
I am fairly new to RL and I compete in AWS DeepRacer student league. The main task there is to create a reward function. All the hyperparameters and action space are fixed. So far, I know how to shape ...
2
votes
1
answer
732
views
How to deal with small reward values
In my environment rewards are generally small, e.g. [-0.01, 0.01]. My concern is that small reward values might get dominated or distorted by the noise during the training. Does it make sense to scale ...
- 341
0
votes
0
answers
30
views
Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ in terms of $p$ (3.4) and $r$ (3.5) [duplicate]
I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:
Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ ...
- 145
4
votes
1
answer
151
views
$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?
I would like to solve the first question of Exercise 3.19 from Sutton and Barto:
Exercise 3.19 The value of an action, $q_{\pi}(s, a)$, depends on the expected next reward and
the expected sum of the ...
- 145
1
vote
1
answer
245
views
Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?
The creation of negative rewards leads to the chance of Q-values being negative. However, networks with relu or sigmoid activations, just cannot predict negative values. This will lead to a case where ...
- 608
0
votes
0
answers
206
views
How to solve a reinforcement learning problem with a stochastic reward function?
In a discrete time system, an environment has an unknown reward probability $p(r|s,a)$. However, the transition probability $p(s'\mid s,a)$ is deterministic.
In my case, the reward for the same action ...
2
votes
0
answers
252
views
How to solve a reinforcement learning problem with changing rewards?
I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for ...
0
votes
0
answers
1k
views
PPO: how to scale rewards
I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried ...
3
votes
1
answer
323
views
Why does the average-reward estimator for continuing tasks use the TD error?
In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q_\pi(s,a)$ and value function $v_\pi(s)$ ...
- 106
2
votes
1
answer
611
views
Why is training longer not better in reinforcement learning?
I have trained an RL agent (PPO) for 6 million steps to solve the OpenAI gym LunarLander-v2. Surprisingly, the agent performs best already after 320K steps and is getting worse after that.
In the ...
- 213
1
vote
2
answers
911
views
What is the difference between a policy and rewards?
I don't understand the difference between a policy and rewards. Sure, a policy tells us what to do, but isn't the output of a neural network trained on rewards basically a policy (i.e. choose the ...
1
vote
0
answers
652
views
What is commonly done for standardization/normalization of the targets in Deep Q-Learning?
I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
- 55
1
vote
0
answers
92
views
Normalisation of reward function
Problem
Currently, I have some problems defining a reward function for my RL project and mainly with how to normalise the score such that the highest possible score for all instances of the ...
- 31
7
votes
2
answers
7k
views
What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?
In Deep Reinforcement Learning (DRL) I am having difficulties in understanding the difference between a Loss function, a reward/penalty and the integration of both in DRL.
Loss function: Given an ...
- 185
0
votes
0
answers
196
views
How to pass the rewards in zero-sum multiplayer context when using REINFORCE?
Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm.
I have doubts about passing reward values in ...
- 3,612
1
vote
0
answers
58
views
Generative systems based on Schmidhuber's compression framework
In Driven by Compression Progress:
A Simple Principle Explains Essential Aspects of
Subjective Beauty, Novelty, Surprise,
Interestingness, Attention, Curiosity, Creativity,
Art, Science, Music, Jokes ...
- 121
0
votes
1
answer
857
views
Is it a bad practice to use cumulative rewards in reinforcement learning
I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over ...
- 69
1
vote
1
answer
357
views
How are rewards calculated for episodic tasks like playing chess or tic-tac-toe?
I am new to Reinforcement Learning and trying to understand the concept of reaping rewards during episodic tasks. I think in games like tic-tac-toe, rewards will be in terms of a win or lose. But does ...
- 11
1
vote
1
answer
80
views
In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?
I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the ...
- 221
0
votes
0
answers
470
views
How to normalize rewards in REINFORCE?
I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to ...
- 121
3
votes
1
answer
222
views
How do I represent sample efficiency of RL rewards in mathematical notation?
I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
- 31
1
vote
2
answers
1k
views
What is the difference between a reward and a value for a given state?
I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I ...
- 57
6
votes
1
answer
145
views
Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?
Say I've got two Markov Decision Processes (MDPs):
$$\mathcal{M_0} = (\mathcal{S}, \mathcal{A}, P, R_0),\quad\text{and}\quad\mathcal{M}_1 = (\mathcal{S}, \mathcal{A}, P, R_1)$$
Both have the same set ...
- 2,426
2
votes
1
answer
466
views
Difference in UCB performance when scaling the rewards
I notice the following behavior when running experiments with $\epsilon$-greedy and UCB1. If the reward is kept binary (0 or 1) both algorithm's performances are on par with each other. However, if I ...
- 223
0
votes
1
answer
155
views
Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning
I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0.
However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) ...
- 371
2
votes
0
answers
68
views
What is the dimensionality of these derivatives in the paper "Active Learning for Reward Estimation in Inverse Reinforcement Learning"?
I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning.
I'm specifically referring to section 2.3 of the paper.
Let's ...
- 141
2
votes
1
answer
99
views
Is there multi-agent reinforcement learning model in which (some of the) reward is given by other agent and not by the external environment?
The traditional setting of multiagent reinforcement learning (MARL) is the mode in which there is set of agents and external environment. And the reward is given to each agent - individually or ...
- 833
3
votes
1
answer
146
views
How do we derive the expression for average reward setting in continuing tasks?
In the average reward setting we have:
$$r(\pi)\doteq \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{t=1}^{h}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$
$$r(\pi)\doteq \lim_{t\rightarrow\infty}\mathbb{E}[R_{t}...
- 147
3
votes
1
answer
3k
views
Why do my rewards reduce after extensive training using D3QN?
I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually ...
- 608
4
votes
2
answers
663
views
Why is regret so defined in MABs?
Consider a multi-armed bandit(MAB). There are $k$ arms, with reward distributions $R_i$ where $1 \leq i \leq k$. Let $\mu_i$ denote the mean of the $i^{th}$ distribution.
If we run the multi-armed ...
- 1,131
1
vote
0
answers
190
views
Why would DDPG with Hindsight Experience Replay not converge?
I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
- 125
3
votes
1
answer
250
views
How can I fix jerky movement in a continuous action space
I am training an agent to do object avoidance. The agent has control over its steering angle and its speed. The steering angle and speed are normalized in a $[−1,1]$ range, where the sign encodes ...
- 65
3
votes
1
answer
305
views
How do I design the rewards and penalties for an agent whose goal it is to explore a map
I am trying to train an agent to explore an unknown two-dimensional map while avoiding circular obstacles (with varying radii). The agent has control over its steering angle and its speed. The ...
- 65
2
votes
0
answers
123
views
Should I use the discounted average reward as objective in a finite-horizon problem?
I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
- 121
5
votes
2
answers
191
views
How can we prevent AGI from doing drugs?
I recently read some introductions to AI alignment, AIXI and decision theory things.
As far as I understood, one of the main problems in AI alignment is how to define a utility function well, not ...
- 153
12
votes
3
answers
2k
views
Why is the reward in reinforcement learning always a scalar?
I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
- 223
4
votes
3
answers
1k
views
Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem?
Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem?
For example, you want to train a DQN agent in an environment, and you want to know what the highest ...
user38696
2
votes
1
answer
3k
views
Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking ...
- 177
3
votes
1
answer
810
views
How is the reward in reinforcement learning different from the label in supervised learning problems?
How is the notion of immediate reward used in the reinforcement learning different from the notion of a label we find in the supervised learning problems?
- 67
6
votes
2
answers
2k
views
Why does shifting all the rewards have a different impact on the performance of the agent?
I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
- 61
2
votes
2
answers
2k
views
How do I calculate the return given the discount factor and a sequence of rewards?
I know that $G_t = R_{t+1} + G_{t+1}$.
Suppose $\gamma = 0.9$ and the reward sequence is $R_1 = 2$ followed by an infinite sequence of $7$s. What is the value of $G_0$?
As it's infinite, how can we ...
- 21