Questions tagged [rewards]

For questions related to the concept of reward, for example, in the context of reinforcement learning and Markov decision processes.

Filter by
Sorted by
Tagged with
0
votes
1answer
39 views

Is it a bad practice to use cumulative rewards in reinforcement learning

I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over ...
1
vote
1answer
33 views

How are rewards calculated for episodic tasks like playing chess or tic-tac-toe?

I am new to Reinforcement Learning and trying to understand the concept of reaping rewards during episodic tasks. I think in games like tic-tac-toe, rewards will be in terms of a win or lose. But does ...
1
vote
1answer
39 views

In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?

I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the ...
0
votes
0answers
72 views

How to normalize rewards in REINFORCE?

I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to ...
0
votes
0answers
19 views

Multi-agent policy gradient, 1 total reward instead of reward in each step, 2 changing action space

I am new in reinforcement learning and not sure I have the right understanding of multi-agent policy gradient. 1, in my question, each agent has its own action space. When doing the sampling, for each ...
2
votes
1answer
57 views

How do I represent sample efficiency of RL rewards in mathematical notation?

I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
1
vote
2answers
193 views

What is the difference between a reward and a value for a given state?

I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I ...
6
votes
1answer
68 views

Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?

Say I've got two Markov Decision Processes (MDPs): $$\mathcal{M_0} = (\mathcal{S}, \mathcal{A}, P, R_0),\quad\text{and}\quad\mathcal{M}_1 = (\mathcal{S}, \mathcal{A}, P, R_1)$$ Both have the same set ...
2
votes
1answer
84 views

Difference in UCB performance when scaling the rewards

I notice the following behavior when running experiments with $\epsilon$-greedy and UCB1. If the reward is kept binary (0 or 1) both algorithm's performances are on par with each other. However, if I ...
0
votes
0answers
42 views

Double DQN backpropagation of negative final rewards?

My problem is that in my Double DQN model, negative final rewards are not being backpropagated into action Q-values, and so some Q-values are positive, when they should be negative, and hence ...
0
votes
1answer
35 views

Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning

I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0. However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) ...
2
votes
0answers
60 views

What is the dimensionality of these derivatives in the paper "Active Learning for Reward Estimation in Inverse Reinforcement Learning"?

I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning. I'm specifically referring to section 2.3 of the paper. Let's ...
2
votes
1answer
39 views

Is there multi-agent reinforcement learning model in which (some of the) reward is given by other agent and not by the external environment?

The traditional setting of multiagent reinforcement learning (MARL) is the mode in which there is set of agents and external environment. And the reward is given to each agent - individually or ...
2
votes
1answer
59 views

How do we derive the expression for average reward setting in continuing tasks?

In the average reward setting we have: $$r(\pi)\doteq \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{t=1}^{h}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$ $$r(\pi)\doteq \lim_{t\rightarrow\infty}\mathbb{E}[R_{t}...
1
vote
1answer
243 views

Why do my rewards reduce after extensive training using D3QN?

I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually ...
4
votes
2answers
153 views

Why is regret so defined in MABs?

Consider a multi-armed bandit(MAB). There are $k$ arms, with reward distributions $R_i$ where $1 \leq i \leq k$. Let $\mu_i$ denote the mean of the $i^{th}$ distribution. If we run the multi-armed ...
1
vote
0answers
71 views

Why would DDPG with Hindsight Experience Replay not converge?

I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
3
votes
1answer
112 views

How can I fix jerky movement in a continuous action space

I am training an agent to do object avoidance. The agent has control over its steering angle and its speed. The steering angle and speed are normalized in a $[−1,1]$ range, where the sign encodes ...
2
votes
1answer
55 views

How do I design the rewards and penalties for an agent whose goal it is to explore a map

I am trying to train an agent to explore an unknown two-dimensional map while avoiding circular obstacles (with varying radii). The agent has control over its steering angle and its speed. The ...
2
votes
0answers
66 views

Should I use the discounted average reward as objective in a finite-horizon problem?

I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
4
votes
2answers
130 views

How can we prevent AGI from doing drugs?

I recently read some introductions to AI alignment, AIXI and decision theory things. As far as I understood, one of the main problems in AI alignment is how to define a utility function well, not ...
12
votes
3answers
623 views

Why is the reward in reinforcement learning always a scalar?

I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
4
votes
3answers
514 views

Upper limit to the maximum cumulative reward in a deep reinforcement learning problem

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem? For example you want to train an DQN agent in an environment and you want to know what is the highest ...
2
votes
1answer
583 views

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking ...
3
votes
1answer
214 views

How is the reward in reinforcement learning different from the label in supervised learning problems?

How is the notion of immediate reward used in the reinforcement learning different from the notion of a label we find in the supervised learning problems?
4
votes
1answer
406 views

Why does shifting all the rewards have a different impact on the performance of the agent?

I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
1
vote
2answers
308 views

How do I calculate the return given the discount factor and a sequence of rewards?

I know that $G_t = R_{t+1} + G_{t+1}$. Suppose $\gamma = 0.9$ and the reward sequence is $R_1 = 2$ followed by an infinite sequence of $7$s. What is the value of $G_0$? As it's infinite, how can we ...
1
vote
0answers
43 views

Does importance sampling for off-policy estimation also apply to the case of negative rewards?

Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and ...
3
votes
1answer
630 views

Non-differentiable reward function to update a neural network

In Reinforcement Learning, when reward function is not differentiable, a policy gradient algorithm is used to update the weights of a network. In the paper Neural Architecture Search with ...
4
votes
1answer
460 views

If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?

I'm trying to solve exercise 3.11 from the book Sutton and Barto's book (2nd edition) Exercise 3.11 If the current state is $S_t$ , and actions are selected according to a stochastic policy $\pi$, ...
1
vote
1answer
78 views

When discounted MAB is useful?

Many of multi-armed bandit(MAB) algorithms are used when the total reward is the sum of all rewards. However, in RL, the discounted reward is mainly used. Why is the discounted reward not prevailing ...
4
votes
2answers
645 views

Is there any difference between reward and return in reinforcement learning?

I am reading Sutton and Barto's book on reinforcement learning. I thought that reward and return were the same things. However, in Section 5.6 of the book, 3rd line, first paragraph, it is written: ...
2
votes
1answer
173 views

Can the agent wait until the end of the episode to determine the reward in SARSA?

From Sutton and Barto's book Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 99), the following definition for first-visit MC prediction, for estimating $V \sim V_\pi$ is ...
2
votes
1answer
501 views

How do you manage negative rewards in policy gradients?

This old question has no definitive answer yet, that's why I am asking it here again. I also asked this same question here. If I'm doing policy gradient in Keras, using a loss of the form: ...
3
votes
1answer
146 views

Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

I have implemented a simple Q-learning algorithm to minimize a cost function by setting the reward to the inverse of the cost of the action taken by the agent. The algorithm converges nicely, but ...
1
vote
0answers
43 views

State-of-the-art algorithms not working on a custom RL environment

I'm trying to train a RL agent on a custom, highly stochastic environment (MDP). In order to do so I'm using existing implementations of state-of-the-art RL algorithms as provided by Stable Baselines. ...
1
vote
1answer
104 views

How do you know if an agent has learnt its environment in reinforcement learning?

I'm new to reinforcement learning and trying to understand it. If you train an agent using a reinforcement learning algorithm (discrete or continuous) on an environment (real or simulated), then how ...
3
votes
1answer
42 views

Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?

I am learning RL for the first time. It may be naive, but it is a bit odd to grasp this idea that, if the goal of RL is to maximize the expected return, then shouldn't the expected return be ...
1
vote
1answer
82 views

Which reward function works for recommendation systems using knowledge graphs?

I've been reading this paper on recommendation systems using reinforcement learning (RL) and knowledge graphs (KGs). To give some background, the graph has several (finitely many) entities, of which ...
3
votes
1answer
181 views

How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?

I'm building an agent for a racing game. In this game, there is a randomized map where there are speed boosts for the player to pick up and obstacles that act to slow the player down. The goal of the ...
3
votes
1answer
226 views

Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity

I'm working on a RL problem with the following properties: The rewards are extremely sparse i.e. all rewards are 0 except the terminal non-zero reward. Ideally I would not use any reward engineering ...
1
vote
1answer
62 views

Can optimizing for immediate reward result in a policy maximizing the return?

The goal of a reinforcement learning agent is to maximize the expected return which is often a discounted sum of future rewards. The return indeed is a very noisy random variable as future rewards ...
2
votes
0answers
35 views

How should I design a reward function for a NLP problem where two models interoperate?

I would like to design a reward function. I am training two models from the first model that classify set of texts (paragraphs and keywords) and I also got some hidden states. The second model is ...
0
votes
1answer
72 views

What is the best measurement for how good an action of a reinforcement learning agent really is?

Even when we get a valuable reward signal after every single action, this immediate reward only approximates the short term goodness of the action. To consider the long term effect of an action, we ...
4
votes
2answers
388 views

Formula for expected rewards for state–action–next-state triples as a three-argument function

While reading about reinforcement learning, I have come across the following expression for expected rewards in terms of a summation, the denominator of which I am not able to account for. The ...
3
votes
0answers
61 views

How does normalization of the inputs work in the context of PPO?

What does the normalization of the inputs mean in the context of PPO? At each time step of an episode, I only know the values of this time step and of the previous ones, if I take track of them. This ...
2
votes
0answers
34 views

How was the DQN trained to play many games?

Some people claim that DQN was used to play many Atari games. But what actually happened? Was DQN trained only once (with some data from all games) or was it trained separately for each game? What was ...
1
vote
1answer
107 views

In RL, if I assign the rewards for better positional play, the algorithm is learning nothing?

I'm creating an RL application for the game Connect Four. If I tell the algorithm which moves/token positions will receive greater rewards, surely it's not actually learning anything; it's just a ...
2
votes
0answers
29 views

How should I define the reward function for the Connect Four game?

I'm creating an RL application for the game Connect Four. I've researched the different strategies for the game and which positions are more favourable to lead to a win. Should I be assigning ...
1
vote
0answers
33 views

How can I find the appropriate reward value for my reinforcement learning problem?

I am wondering how can I find the appropriate reward value for each specific problem. I know this is a highly empirical process, but I am sure that the value is not set totally at random. I want to ...