10
votes
Accepted
Why does the "reward to go" trick in policy gradient methods work?
An important thing we're going to need is what is called the "Expected Grad-Log-Prob Lemma here" (proof included on that page), which says that (for any $t$):
$$\mathbb{E}_{\tau \sim \pi_{\...
- 10.1k
8
votes
Accepted
Why is the reward in reinforcement learning always a scalar?
If you have multiple types of rewards (say, R1 and R2), then it is no longer clear what would be the optimal way to act: it can happen that one way of acting would maximize R1 and another way would ...
- 231
7
votes
Accepted
What is the difference between expected return and value function?
There is a strong relationship between a value function and a return. Namely that a value function calculates the expected return from being in a certain state, or taking a specific action in a ...
- 30.4k
7
votes
How do I handle negative rewards in policy gradients with the cross-entropy loss function?
It depends on your loss function, but you probably need to tweak it.
If you are using an update rule like loss = -log(probabilities) * reward, then your loss is ...
- 171
7
votes
Why is the reward in reinforcement learning always a scalar?
Rather than the survey by Liu et al. recommended above, I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a ...
- 121
7
votes
Accepted
What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?
1. Question: The difference between loss and reward/penalty
So I see both the loss function and the reward/penalty are the quantitative way of measuring the output/action and making the model to ...
- 1,536
6
votes
Accepted
Is there any difference between reward and return in reinforcement learning?
Return refers to the total discounted reward, starting from the current timestep.
- 1,131
6
votes
Why is the reward in reinforcement learning always a scalar?
Markov decision problems are usually defined with a reward function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and in these cases the rewards are expected to be scalar real values. This ...
- 552
6
votes
Accepted
Why do my rewards reduce after extensive training using D3QN?
It is not 100% clear, but this seems like an instance of catastrophic forgetting. This is something that often impacts reinforcement learning.
I have answered a very similar question on Data Science ...
- 30.4k
5
votes
What would motivate a machine?
This is an interesting question actually.
There's a quite realistic idea about "where can the curiosity originate from" in the book "On intelligence" written by Jeff Hawkins and Sandra Blakeslee.
It'...
- 109
5
votes
Accepted
What would motivate a machine?
The current method to implement motivation is some kind of artificial reward. Deepmind's DQN for example is driven by the score of the game. The higher the score, the better. The AI learns to adjust ...
- 1,684
5
votes
Accepted
Should RL rewards diminish over time?
RL agents - implemented correctly - do not take previous rewards into account when making decisions. For instance value functions only assess potential future reward. The state value or expected ...
- 30.4k
5
votes
Why cannot an AI agent adjust the reward function directly?
Why do both approaches prevent the AI agent from changing its reward function at will?
In RL for optimal control, the reward function is part of the problem formulation. That is, it describes the ...
- 30.4k
4
votes
Reinforcement Learning with long term rewards and fixed states and actions
You don't need to have a reward on every single timestep, reward at the end is enough. Reinforcement learning can deal with temporal credit assignment problem, all algorithms are designed to work with ...
- 2,336
4
votes
In RL, if I assign the rewards for better positional play, the algorithm is learning nothing?
What you are proposing is closer to a heuristic for searching than a reward for RL. This is a blurred line, but generally if you start analysing the problem yourself, breaking it down into components ...
- 30.4k
4
votes
Accepted
Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?
Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in ...
- 30.4k
4
votes
Accepted
If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?
First note that $\mathbb{E}[R_{t+1} |S_t=s] = \sum_{s',r}rm(s',r|s)$ where $m(\cdot)$ is the mass function for the joint distribution of $S_{t+1},R_{t+1}$.
If you are currently in state $S_t$ and we ...
- 4,675
4
votes
Non-differentiable reward function to update a neural network
I cannot wrap my head around the concept of accuracy as a non-differentiable reward function. Do we need to find the function and then check if it is mathematically non-differentiable?
In ...
- 30.4k
4
votes
Accepted
Why is regret so defined in MABs?
In short, you don't regret your bad luck that you could do nothing about, you regret your bad choices that you could have done something about if only you knew.
The point of regret as a metric ...
- 30.4k
4
votes
Accepted
Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?
I believe the claim is true. Here is my attempt at a proof.
Let us consider the optimal infinite horizon value function $V_\alpha^*$ of $\mathcal{M}_\alpha$ at an arbitrary state $s \in S$.
The value $...
- 580
4
votes
Accepted
What is the difference between a reward and a value for a given state?
Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for ...
- 30.4k
4
votes
Accepted
How to deal with small reward values
The numbers that a value-based neural network will predict are usually based on expected returns (sum of rewards by end of an episode, or a discounted infinite sum), although in some cases they might ...
- 30.4k
3
votes
Accepted
How to evaluate an RL algorithm when used in a game?
When you want to compare Reinforcement Learning algorithms, you might want to compare the average rewards they generate and how fast and close they get to the optimal policy. However, in the case of ...
- 365
3
votes
Accepted
Are there any reliable ways of modifying the reward function to make the rewards less sparse?
Doing something like the dense, distance-based reward signal you propose is possible... but you have to do it very carefully. If you're not careful, and do it in a naive manner, you are likely to ...
- 10.1k
3
votes
Accepted
Reinforcement Learning with long term rewards and fixed states and actions
You are describing a straightforward Markov Decision Process that could be solved by almost any Reinforcement Learning algorithm.
I have read a lot about RL algorithms, that update the action-value ...
- 30.4k
3
votes
What would motivate a machine?
I asked professor Richard Sutton a similar question, in the first lecture of the reinforcement learning course. It seems that there are different ways to motivate the machine. In fact, machine ...
- 251
3
votes
How do I handle negative rewards in policy gradients with the cross-entropy loss function?
The cross-entropy loss will always be positive because the probability is in the range $[0, 1]$, so $-ln(p)$ will always be positive.
- 31
3
votes
Is my interpretation of the return correct?
Your table is almost correct. It is a minor difference, you should not have a $R_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R_1$ (a result of ...
- 30.4k
3
votes
Accepted
How should I take into consideration the number of steps in the reward function?
How would you implement this "Number of Steps" cost?
What the paper is referring to is the reward discounting process which is a standard way of formulating RL problems, either continuous ...
- 30.4k
3
votes
Is there a good ratio between the positive and negative rewards in reinforcement learning?
It usually does not matter, but I'm sure there are situations where it could matter. In theory, if a reward for good behavior is higher than the rewards for bad behavior, then the neural network will ...
- 53
Only top scored, non community-wiki answers of a minimum length are eligible