Questions tagged [rewards]
For questions related to the concept of reward, for example, in the context of reinforcement learning and Markov decision processes. For questions related to reward functions, reward design, reward shaping, reward hacking, etc., there are also those specific tags, so you use them instead of this generic one, unless your question is also about the concept of reward.
120
questions
0
votes
0
answers
26
views
Penalizing invalid action and ending episode early causes agent to end episode with invalid action in a non-goal state?
Domain: start at initial position, navigate to goal-position with N,S,E,W actions.
Algorithm: PPO
Libs: custom gym-env, stable baselines3
Penalties:
valid step: -1 (promote shortest path)
INvalid step:...
0
votes
0
answers
31
views
How to evaluate the performance of off-line & model-free reinforcement leaning?
I'm currently studying on off-line reinforcement learning (RL) and trying to utilize it for medical data. Because it seemed hard to develop well-performing environment model, I decided to adopt model-...
- 1
1
vote
0
answers
145
views
MDP with a non-markovian reward function?
I have set up a RL environment and it converges to a decent solution when using a reward function:
$R(s_t,a_t) = fenv(s_t, a_t)$ , where $fenv$ is the environment dynamics.
Now, i want to change the ...
- 11
1
vote
1
answer
94
views
How to normalizing various elements of the reward function?
Suppose I have a reward function $R$ that I wish to penalize w.r.t two distinct phenomenons $A$ and $B$. $A$, for example, could represent the phenomenon of the state not crossing some boundary $[s_1,...
- 361
0
votes
0
answers
41
views
Frozen baseline for policy gradient rewards
I have a continuous reinforcement learning problem for which I use policy gradients and I use a baseline to decrease the variance of the gradients.
The baseline that I used is the moving average of ...
- 1
0
votes
0
answers
55
views
How do I improve the reward of policy gradient network when multiple states and actions exist per time step?
I am working on a project, in which I'm using a policy gradient algorithm (REINFORCE) to select the best cleaning method/methods for erroneous samples in tabular datasets.
The details are as follows.
...
- 1
0
votes
1
answer
114
views
Should the concept of discounted rewards result in multiple arrays per episode in RL?
Note that I'm coming from mostly only working with the REINFORCE algorithm, but I've typically seen discounted rewards calculated in a way that looks like below:
Say you have a reward array of length <...
- 89
0
votes
0
answers
82
views
Shaping reward so that it maximizes multiple components together
I am fairly new to RL and I compete in AWS DeepRacer student league. The main task there is to create a reward function. All the hyperparameters and action space are fixed. So far, I know how to shape ...
0
votes
0
answers
242
views
Reward shaping for an autonomous driving car (AWS DeepRacer)
From past 2 months, I am competing in AWS Deepracer student league. I was new to RL but had some knowledge in supervised and unsupervised learning. In the league the hyperparameters and action space ...
0
votes
0
answers
61
views
When use positive or negative rewards in reinforcement learning? Is there anything in literature?
Let's say I can design a reward as function of a distance $d>0$ from the target in 2 ways: $r=\frac{1}{1+d}$ or $r=-d$. The first is defined in $(0,1]$ the second in $(-\infty,0]$. I would expect ...
- 41
0
votes
0
answers
27
views
How to design a DQN that needs to self-play with very delayed rewards?
I got a single-player card game (comparable to BlackJack) in which rewards are VERY delayed. On average a player can estimate the effect of his move fifteen steps deep. As there are 52 cards to play ...
- 41
0
votes
0
answers
27
views
Q-learning: In tic-tac-toe, how to choose rewards values?
| I'm experimenting for first times in neural networks and Q-learning.
In tic-tac-toe game, each move has 5 possible esits
invalid move
valid move, match is still open
draft
you lose
you win
...
- 101
2
votes
1
answer
532
views
How to deal with small reward values
In my environment rewards are generally small, e.g. [-0.01, 0.01]. My concern is that small reward values might get dominated or distorted by the noise during the training. Does it make sense to scale ...
- 331
0
votes
0
answers
29
views
Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ in terms of $p$ (3.4) and $r$ (3.5) [duplicate]
I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:
Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ ...
- 95
4
votes
1
answer
101
views
$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?
I would like to solve the first question of Exercise 3.19 from Sutton and Barto:
Exercise 3.19 The value of an action, $q_{\pi}(s, a)$, depends on the expected next reward and
the expected sum of the ...
- 95
1
vote
1
answer
165
views
Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?
The creation of negative rewards leads to the chance of Q-values being negative. However, networks with relu or sigmoid activations, just cannot predict negative values. This will lead to a case where ...
- 518
0
votes
0
answers
180
views
How to solve a reinforcement learning problem with a stochastic reward function?
In a discrete time system, an environment has an unknown reward probability $p(r|s,a)$. However, the transition probability $p(s'\mid s,a)$ is deterministic.
In my case, the reward for the same action ...
2
votes
0
answers
208
views
How to solve a reinforcement learning problem with changing rewards?
I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for ...
0
votes
0
answers
770
views
PPO: how to scale rewards
I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried ...
2
votes
1
answer
255
views
Why does the average-reward estimator for continuing tasks use the TD error?
In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q_\pi(s,a)$ and value function $v_\pi(s)$ ...
- 86
2
votes
1
answer
473
views
Why is training longer not better in reinforcement learning?
I have trained an RL agent (PPO) for 6 million steps to solve the OpenAI gym LunarLander-v2. Surprisingly, the agent performs best already after 320K steps and is getting worse after that.
In the ...
- 213
1
vote
2
answers
610
views
What is the difference between a policy and rewards?
I don't understand the difference between a policy and rewards. Sure, a policy tells us what to do, but isn't the output of a neural network trained on rewards basically a policy (i.e. choose the ...
1
vote
0
answers
455
views
What is commonly done for standardization/normalization of the targets in Deep Q-Learning?
I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
- 55
1
vote
0
answers
64
views
Normalisation of reward function
Problem
Currently, I have some problems defining a reward function for my RL project and mainly with how to normalise the score such that the highest possible score for all instances of the ...
- 31
5
votes
2
answers
4k
views
What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?
In Deep Reinforcement Learning (DRL) I am having difficulties in understanding the difference between a Loss function, a reward/penalty and the integration of both in DRL.
Loss function: Given an ...
- 165
0
votes
0
answers
155
views
How to pass the rewards in zero-sum multiplayer context when using REINFORCE?
Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm.
I have doubts about passing reward values in ...
- 3,531
1
vote
0
answers
49
views
Generative systems based on Schmidhuber's compression framework
In Driven by Compression Progress:
A Simple Principle Explains Essential Aspects of
Subjective Beauty, Novelty, Surprise,
Interestingness, Attention, Curiosity, Creativity,
Art, Science, Music, Jokes ...
- 121
0
votes
1
answer
616
views
Is it a bad practice to use cumulative rewards in reinforcement learning
I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over ...
- 69
1
vote
1
answer
276
views
How are rewards calculated for episodic tasks like playing chess or tic-tac-toe?
I am new to Reinforcement Learning and trying to understand the concept of reaping rewards during episodic tasks. I think in games like tic-tac-toe, rewards will be in terms of a win or lose. But does ...
- 11
1
vote
1
answer
69
views
In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?
I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the ...
- 221
0
votes
0
answers
370
views
How to normalize rewards in REINFORCE?
I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to ...
- 121
3
votes
1
answer
203
views
How do I represent sample efficiency of RL rewards in mathematical notation?
I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
- 31
1
vote
2
answers
1k
views
What is the difference between a reward and a value for a given state?
I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I ...
- 57
6
votes
1
answer
122
views
Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?
Say I've got two Markov Decision Processes (MDPs):
$$\mathcal{M_0} = (\mathcal{S}, \mathcal{A}, P, R_0),\quad\text{and}\quad\mathcal{M}_1 = (\mathcal{S}, \mathcal{A}, P, R_1)$$
Both have the same set ...
- 2,380
2
votes
1
answer
349
views
Difference in UCB performance when scaling the rewards
I notice the following behavior when running experiments with $\epsilon$-greedy and UCB1. If the reward is kept binary (0 or 1) both algorithm's performances are on par with each other. However, if I ...
- 223
0
votes
1
answer
134
views
Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning
I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0.
However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) ...
- 371
2
votes
0
answers
67
views
What is the dimensionality of these derivatives in the paper "Active Learning for Reward Estimation in Inverse Reinforcement Learning"?
I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning.
I'm specifically referring to section 2.3 of the paper.
Let's ...
- 141
2
votes
1
answer
87
views
Is there multi-agent reinforcement learning model in which (some of the) reward is given by other agent and not by the external environment?
The traditional setting of multiagent reinforcement learning (MARL) is the mode in which there is set of agents and external environment. And the reward is given to each agent - individually or ...
- 813
3
votes
1
answer
137
views
How do we derive the expression for average reward setting in continuing tasks?
In the average reward setting we have:
$$r(\pi)\doteq \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{t=1}^{h}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$
$$r(\pi)\doteq \lim_{t\rightarrow\infty}\mathbb{E}[R_{t}...
- 147
3
votes
1
answer
2k
views
Why do my rewards reduce after extensive training using D3QN?
I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually ...
- 518
4
votes
2
answers
530
views
Why is regret so defined in MABs?
Consider a multi-armed bandit(MAB). There are $k$ arms, with reward distributions $R_i$ where $1 \leq i \leq k$. Let $\mu_i$ denote the mean of the $i^{th}$ distribution.
If we run the multi-armed ...
- 1,091
1
vote
0
answers
162
views
Why would DDPG with Hindsight Experience Replay not converge?
I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
- 105
3
votes
1
answer
209
views
How can I fix jerky movement in a continuous action space
I am training an agent to do object avoidance. The agent has control over its steering angle and its speed. The steering angle and speed are normalized in a $[−1,1]$ range, where the sign encodes ...
- 65
3
votes
1
answer
210
views
How do I design the rewards and penalties for an agent whose goal it is to explore a map
I am trying to train an agent to explore an unknown two-dimensional map while avoiding circular obstacles (with varying radii). The agent has control over its steering angle and its speed. The ...
- 65
2
votes
0
answers
112
views
Should I use the discounted average reward as objective in a finite-horizon problem?
I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
- 121
4
votes
2
answers
176
views
How can we prevent AGI from doing drugs?
I recently read some introductions to AI alignment, AIXI and decision theory things.
As far as I understood, one of the main problems in AI alignment is how to define a utility function well, not ...
- 143
12
votes
3
answers
1k
views
Why is the reward in reinforcement learning always a scalar?
I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
- 223
4
votes
3
answers
965
views
Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem?
Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem?
For example, you want to train a DQN agent in an environment, and you want to know what the highest ...
user38696
2
votes
1
answer
2k
views
Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking ...
- 167
3
votes
1
answer
650
views
How is the reward in reinforcement learning different from the label in supervised learning problems?
How is the notion of immediate reward used in the reinforcement learning different from the notion of a label we find in the supervised learning problems?
- 67