Skip to main content
Share Your Experience: Take the 2024 Developer Survey
12 votes
Accepted

Why does the "reward to go" trick in policy gradient methods work?

An important thing we're going to need is what is called the "Expected Grad-Log-Prob Lemma here" (proof included on that page), which says that (for any $t$): $$\mathbb{E}_{\tau \sim \pi_{\...
Dennis Soemers's user avatar
  • 10.3k
9 votes
Accepted

Why is the reward in reinforcement learning always a scalar?

If you have multiple types of rewards (say, R1 and R2), then it is no longer clear what would be the optimal way to act: it can happen that one way of acting would maximize R1 and another way would ...
present's user avatar
  • 241
8 votes

Why is the reward in reinforcement learning always a scalar?

Rather than the survey by Liu et al. recommended above, I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a ...
Peter Vamplew's user avatar
7 votes
Accepted

What is the difference between expected return and value function?

There is a strong relationship between a value function and a return. Namely that a value function calculates the expected return from being in a certain state, or taking a specific action in a ...
Neil Slater's user avatar
  • 32.4k
7 votes

How do I handle negative rewards in policy gradients with the cross-entropy loss function?

It depends on your loss function, but you probably need to tweak it. If you are using an update rule like loss = -log(probabilities) * reward, then your loss is ...
Tahlor's user avatar
  • 171
7 votes

Why is the reward in reinforcement learning always a scalar?

Markov decision problems are usually defined with a reward function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and in these cases the rewards are expected to be scalar real values. This ...
Hai Nguyen's user avatar
7 votes
Accepted

What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?

1. Question: The difference between loss and reward/penalty So I see both the loss function and the reward/penalty are the quantitative way of measuring the output/action and making the model to ...
Chillston's user avatar
  • 1,748
6 votes
Accepted

Is there any difference between reward and return in reinforcement learning?

Return refers to the total discounted reward, starting from the current timestep.
stoic-santiago's user avatar
6 votes
Accepted

Why do my rewards reduce after extensive training using D3QN?

It is not 100% clear, but this seems like an instance of catastrophic forgetting. This is something that often impacts reinforcement learning. I have answered a very similar question on Data Science ...
Neil Slater's user avatar
  • 32.4k
5 votes

What would motivate a machine?

This is an interesting question actually. There's a quite realistic idea about "where can the curiosity originate from" in the book "On intelligence" written by Jeff Hawkins and Sandra Blakeslee. It'...
Ivan Bogush's user avatar
5 votes
Accepted

What would motivate a machine?

The current method to implement motivation is some kind of artificial reward. Deepmind's DQN for example is driven by the score of the game. The higher the score, the better. The AI learns to adjust ...
Demento's user avatar
  • 1,684
5 votes
Accepted

Should RL rewards diminish over time?

RL agents - implemented correctly - do not take previous rewards into account when making decisions. For instance value functions only assess potential future reward. The state value or expected ...
Neil Slater's user avatar
  • 32.4k
5 votes

Why cannot an AI agent adjust the reward function directly?

Why do both approaches prevent the AI agent from changing its reward function at will? In RL for optimal control, the reward function is part of the problem formulation. That is, it describes the ...
Neil Slater's user avatar
  • 32.4k
5 votes
Accepted

If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?

First note that $\mathbb{E}[R_{t+1} |S_t=s] = \sum_{s',r}rm(s',r|s)$ where $m(\cdot)$ is the mass function for the joint distribution of $S_{t+1},R_{t+1}$. If you are currently in state $S_t$ and we ...
David's user avatar
  • 4,910
4 votes

Reinforcement Learning with long term rewards and fixed states and actions

You don't need to have a reward on every single timestep, reward at the end is enough. Reinforcement learning can deal with temporal credit assignment problem, all algorithms are designed to work with ...
Brale's user avatar
  • 2,406
4 votes

In RL, if I assign the rewards for better positional play, the algorithm is learning nothing?

What you are proposing is closer to a heuristic for searching than a reward for RL. This is a blurred line, but generally if you start analysing the problem yourself, breaking it down into components ...
Neil Slater's user avatar
  • 32.4k
4 votes

Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity

(1) You might want look into RND (Random network distillation) which allows usage of a curiosity-based exploration bonus for the agent as an intrinsic reward. You can use the intrinsic reward to ...
Huan's user avatar
  • 171
4 votes
Accepted

Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in ...
Neil Slater's user avatar
  • 32.4k
4 votes

Non-differentiable reward function to update a neural network

I cannot wrap my head around the concept of accuracy as a non-differentiable reward function. Do we need to find the function and then check if it is mathematically non-differentiable? In ...
Neil Slater's user avatar
  • 32.4k
4 votes
Accepted

Why is regret so defined in MABs?

In short, you don't regret your bad luck that you could do nothing about, you regret your bad choices that you could have done something about if only you knew. The point of regret as a metric ...
Neil Slater's user avatar
  • 32.4k
4 votes
Accepted

Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?

I believe the claim is true. Here is my attempt at a proof. Let us consider the optimal infinite horizon value function $V_\alpha^*$ of $\mathcal{M}_\alpha$ at an arbitrary state $s \in S$. The value $...
mikkola's user avatar
  • 579
4 votes
Accepted

What is the difference between a reward and a value for a given state?

Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for ...
Neil Slater's user avatar
  • 32.4k
4 votes
Accepted

How to deal with small reward values

The numbers that a value-based neural network will predict are usually based on expected returns (sum of rewards by end of an episode, or a discounted infinite sum), although in some cases they might ...
Neil Slater's user avatar
  • 32.4k
4 votes
Accepted

What are reward networks in reinforcement learning?

In RL, reward networks (also called reward models), say $r_\theta$, have the sole role of learning (approximating) the reward function, defined as $r(s,a)$ or $r(s, a,s')$, in which the latter ...
Luca Anzalone's user avatar
3 votes
Accepted

How to evaluate an RL algorithm when used in a game?

When you want to compare Reinforcement Learning algorithms, you might want to compare the average rewards they generate and how fast and close they get to the optimal policy. However, in the case of ...
agold's user avatar
  • 375
3 votes
Accepted

Are there any reliable ways of modifying the reward function to make the rewards less sparse?

Doing something like the dense, distance-based reward signal you propose is possible... but you have to do it very carefully. If you're not careful, and do it in a naive manner, you are likely to ...
Dennis Soemers's user avatar
  • 10.3k
3 votes
Accepted

Reinforcement Learning with long term rewards and fixed states and actions

You are describing a straightforward Markov Decision Process that could be solved by almost any Reinforcement Learning algorithm. I have read a lot about RL algorithms, that update the action-value ...
Neil Slater's user avatar
  • 32.4k
3 votes

What would motivate a machine?

I asked professor Richard Sutton a similar question, in the first lecture of the reinforcement learning course. It seems that there are different ways to motivate the machine. In fact, machine ...
A.Rashad's user avatar
  • 251
3 votes

How do I handle negative rewards in policy gradients with the cross-entropy loss function?

The cross-entropy loss will always be positive because the probability is in the range $[0, 1]$, so $-ln(p)$ will always be positive.
user3711746's user avatar
3 votes

Is my interpretation of the return correct?

Your table is almost correct. It is a minor difference, you should not have a $R_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R_1$ (a result of ...
Neil Slater's user avatar
  • 32.4k

Only top scored, non community-wiki answers of a minimum length are eligible