Questions tagged [rewards]

For questions related to the concept of reward, for example, in the context of reinforcement learning and Markov decision processes. For questions related to reward functions, reward design, reward shaping, reward hacking, etc., there are also those specific tags, so you use them instead of this generic one, unless your question is also about the concept of reward.

Filter by
Sorted by
Tagged with
0
votes
0answers
32 views

How to pass the rewards in zero-sum multiplayer context when using REINFORCE?

Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm. I have doubts about passing reward values in ...
0
votes
0answers
80 views

How to normalize rewards in REINFORCE?

I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to ...
1
vote
0answers
25 views

Generative systems based on Schmidhuber's compression framework

In Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes ...
2
votes
1answer
261 views

How do we know that the algorithm has converged and ensures the highest possible reward?

I started learning about Q table from this blog post Introduction to reinforcement learning and OpenAI Gym, by Justin Francis, which has a line as below - After so many episodes, the algorithm will ...
4
votes
1answer
145 views

What could be the cause of the drop in the reward in A3C?

The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause. The problem has a sparse reward, batch size=2000, <...
2
votes
1answer
61 views

How do I represent sample efficiency of RL rewards in mathematical notation?

I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
-1
votes
1answer
47 views

Is it a bad practice to use cumulative rewards in reinforcement learning

I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over ...
4
votes
1answer
494 views

If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?

I'm trying to solve exercise 3.11 from the book Sutton and Barto's book (2nd edition) Exercise 3.11 If the current state is $S_t$ , and actions are selected according to a stochastic policy $\pi$, ...
12
votes
6answers
2k views

What would motivate a machine?

Currently, within the AI development field, the main focus seems to be on pattern recognition and machine learning. Learning is about adjusting internal variables based on a feedback loop. Maslow's ...
1
vote
1answer
42 views

How are rewards calculated for episodic tasks like playing chess or tic-tac-toe?

I am new to Reinforcement Learning and trying to understand the concept of reaping rewards during episodic tasks. I think in games like tic-tac-toe, rewards will be in terms of a win or lose. But does ...
12
votes
3answers
664 views

Why is the reward in reinforcement learning always a scalar?

I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
1
vote
1answer
41 views

In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?

I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the ...
0
votes
0answers
20 views

Multi-agent policy gradient, 1 total reward instead of reward in each step, 2 changing action space

I am new in reinforcement learning and not sure I have the right understanding of multi-agent policy gradient. 1, in my question, each agent has its own action space. When doing the sampling, for each ...
1
vote
2answers
220 views

What is the difference between a reward and a value for a given state?

I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I ...
6
votes
1answer
71 views

Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?

Say I've got two Markov Decision Processes (MDPs): $$\mathcal{M_0} = (\mathcal{S}, \mathcal{A}, P, R_0),\quad\text{and}\quad\mathcal{M}_1 = (\mathcal{S}, \mathcal{A}, P, R_1)$$ Both have the same set ...
0
votes
1answer
39 views

Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning

I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0. However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) ...
2
votes
1answer
98 views

Difference in UCB performance when scaling the rewards

I notice the following behavior when running experiments with $\epsilon$-greedy and UCB1. If the reward is kept binary (0 or 1) both algorithm's performances are on par with each other. However, if I ...
2
votes
1answer
63 views

Do we have to consider the feasability of an action when defining the reward function of a MDP?

Do we have to consider if (s is given) an action a can lead to s' when defining a reward function? To be more specific: Let's say I have a 1D Map like: |A|B|C|D| ...
0
votes
0answers
47 views

Double DQN backpropagation of negative final rewards?

My problem is that in my Double DQN model, negative final rewards are not being backpropagated into action Q-values, and so some Q-values are positive, when they should be negative, and hence ...
1
vote
1answer
78 views

When discounted MAB is useful?

Many of multi-armed bandit(MAB) algorithms are used when the total reward is the sum of all rewards. However, in RL, the discounted reward is mainly used. Why is the discounted reward not prevailing ...
4
votes
2answers
166 views

Why is regret so defined in MABs?

Consider a multi-armed bandit(MAB). There are $k$ arms, with reward distributions $R_i$ where $1 \leq i \leq k$. Let $\mu_i$ denote the mean of the $i^{th}$ distribution. If we run the multi-armed ...
4
votes
1answer
229 views

How to evaluate an RL algorithm when used in a game?

I'm planning to create a web-based RL board game, and I wondered how I would evaluate the performance of the RL agent. How would I be able to say, "Version X performed better than version Y, as ...
2
votes
0answers
60 views

What is the dimensionality of these derivatives in the paper "Active Learning for Reward Estimation in Inverse Reinforcement Learning"?

I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning. I'm specifically referring to section 2.3 of the paper. Let's ...
10
votes
1answer
3k views

What is the difference between expected return and value function?

I've seen numerous mathematical explanations of reward, value functions $V(s)$, and return functions. The reward provides an immediate return for being in a specific state. The better the reward, the ...
1
vote
1answer
1k views

What is the reward system of reinforcement learning?

Can you describe this reward system in more detail? I understand that the environment sends a signal indicating whether or not the action taken by the agent was 'good' or not, but it seems too simple. ...
2
votes
1answer
43 views

Is there multi-agent reinforcement learning model in which (some of the) reward is given by other agent and not by the external environment?

The traditional setting of multiagent reinforcement learning (MARL) is the mode in which there is set of agents and external environment. And the reward is given to each agent - individually or ...
5
votes
1answer
2k views

Why does the "reward to go" trick in policy gradient methods work?

In the policy gradient method, there's a trick to reduce the variance of policy gradient. We use causality, and remove part of the sum over rewards so that only actions happened after the reward are ...
1
vote
0answers
80 views

Why would DDPG with Hindsight Experience Replay not converge?

I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
1
vote
0answers
43 views

Does importance sampling for off-policy estimation also apply to the case of negative rewards?

Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and ...
4
votes
1answer
462 views

Why does shifting all the rewards have a different impact on the performance of the agent?

I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
2
votes
1answer
536 views

How do you manage negative rewards in policy gradients?

This old question has no definitive answer yet, that's why I am asking it here again. I also asked this same question here. If I'm doing policy gradient in Keras, using a loss of the form: ...
9
votes
2answers
10k views

How do I handle negative rewards in policy gradients with the cross-entropy loss function?

I am using policy gradients in my reinforcement learning algorithm, and occasionally my environment provides a severe penalty (i.e. negative reward) when a wrong move is made. I'm using a neural ...
3
votes
1answer
162 views

Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

I have implemented a simple Q-learning algorithm to minimize a cost function by setting the reward to the inverse of the cost of the action taken by the agent. The algorithm converges nicely, but ...
2
votes
1answer
66 views

How do we derive the expression for average reward setting in continuing tasks?

In the average reward setting we have: $$r(\pi)\doteq \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{t=1}^{h}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$ $$r(\pi)\doteq \lim_{t\rightarrow\infty}\mathbb{E}[R_{t}...
2
votes
1answer
58 views

How do I design the rewards and penalties for an agent whose goal it is to explore a map

I am trying to train an agent to explore an unknown two-dimensional map while avoiding circular obstacles (with varying radii). The agent has control over its steering angle and its speed. The ...
3
votes
1answer
202 views

How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?

I'm building an agent for a racing game. In this game, there is a randomized map where there are speed boosts for the player to pick up and obstacles that act to slow the player down. The goal of the ...
2
votes
1answer
450 views

How should I define the reward function in the case of Connect Four?

I'm using RL to train a Network on the game Connect4. It learns quickly that 4 connected pieces is good. It gets a reward of 1 for this. A zero is rewarded for all other moves. It takes quite a time ...
3
votes
1answer
91 views

Are there any reliable ways of modifying the reward function to make the rewards less sparse?

If I am training an agent to try and navigate a maze as fast as possible, a simple reward would be something like \begin{align} R(\text{terminal}) &= N - \text{time}\ \ , \ \ N \gg \text{...
3
votes
1answer
188 views

How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?

I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (in $[-1, 1]$). To this end, I use a ...
1
vote
1answer
268 views

Why do my rewards reduce after extensive training using D3QN?

I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually ...
2
votes
1answer
66 views

How should I take into consideration the number of steps in the reward function?

I am currently implementing the paper Active Object Localization with Deep Reinforcement Learning in Python. While reading about the reward scheme I came across the following: Finally, the proposed ...
3
votes
1answer
113 views

How can I fix jerky movement in a continuous action space

I am training an agent to do object avoidance. The agent has control over its steering angle and its speed. The steering angle and speed are normalized in a $[−1,1]$ range, where the sign encodes ...
4
votes
2answers
133 views

How can we prevent AGI from doing drugs?

I recently read some introductions to AI alignment, AIXI and decision theory things. As far as I understood, one of the main problems in AI alignment is how to define a utility function well, not ...
2
votes
0answers
69 views

Should I use the discounted average reward as objective in a finite-horizon problem?

I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
3
votes
1answer
127 views

Can recovering a reward function using IRL lead to better policies compared to reward shaping?

I am working on a research project about the different reward functions being used in the RL domain. I have read up on Inverse Reinforcement Learning (IRL) and Reward Shaping (RS). I would like to ...
4
votes
3answers
537 views

Upper limit to the maximum cumulative reward in a deep reinforcement learning problem

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem? For example you want to train an DQN agent in an environment and you want to know what is the highest ...
2
votes
1answer
670 views

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking ...
3
votes
1answer
240 views

How is the reward in reinforcement learning different from the label in supervised learning problems?

How is the notion of immediate reward used in the reinforcement learning different from the notion of a label we find in the supervised learning problems?
1
vote
2answers
370 views

How do I calculate the return given the discount factor and a sequence of rewards?

I know that $G_t = R_{t+1} + G_{t+1}$. Suppose $\gamma = 0.9$ and the reward sequence is $R_1 = 2$ followed by an infinite sequence of $7$s. What is the value of $G_0$? As it's infinite, how can we ...
3
votes
1answer
680 views

Non-differentiable reward function to update a neural network

In Reinforcement Learning, when reward function is not differentiable, a policy gradient algorithm is used to update the weights of a network. In the paper Neural Architecture Search with ...