Hot answers tagged

9 votes
Accepted

Why does the "reward to go" trick in policy gradient methods work?

An important thing we're going to need is what is called the "Expected Grad-Log-Prob Lemma here" (proof included on that page), which says that (for any $t$): $$\mathbb{E}_{\tau \sim \pi_{\...
user avatar
  • 9,379
8 votes
Accepted

Why is the reward in reinforcement learning always a scalar?

If you have multiple types of rewards (say, R1 and R2), then it is no longer clear what would be the optimal way to act: it can happen that one way of acting would maximize R1 and another way would ...
user avatar
  • 231
7 votes
Accepted

What is the difference between expected return and value function?

There is a strong relationship between a value function and a return. Namely that a value function calculates the expected return from being in a certain state, or taking a specific action in a ...
user avatar
  • 23.8k
7 votes

Why is the reward in reinforcement learning always a scalar?

Rather than the survey by Liu et al. recommended above, I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a ...
user avatar
6 votes

How do I handle negative rewards in policy gradients with the cross-entropy loss function?

It depends on your loss function, but you probably need to tweak it. If you are using an update rule like loss = -log(probabilities) * reward, then your loss is ...
user avatar
  • 161
6 votes

Why is the reward in reinforcement learning always a scalar?

Markov decision problems are usually defined with a reward function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and in these cases the rewards are expected to be scalar real values. This ...
user avatar
5 votes

Why cannot an AI agent adjust the reward function directly?

Why do both approaches prevent the AI agent from changing its reward function at will? In RL for optimal control, the reward function is part of the problem formulation. That is, it describes the ...
user avatar
  • 23.8k
5 votes
Accepted

Should RL rewards diminish over time?

RL agents - implemented correctly - do not take previous rewards into account when making decisions. For instance value functions only assess potential future reward. The state value or expected ...
user avatar
  • 23.8k
5 votes

What would motivate a machine?

This is an interesting question actually. There's a quite realistic idea about "where can the curiosity originate from" in the book "On intelligence" written by Jeff Hawkins and Sandra Blakeslee. It'...
user avatar
5 votes
Accepted

What would motivate a machine?

The current method to implement motivation is some kind of artificial reward. Deepmind's DQN for example is driven by the score of the game. The higher the score, the better. The AI learns to adjust ...
user avatar
  • 1,644
4 votes

Reinforcement Learning with long term rewards and fixed states and actions

You don't need to have a reward on every single timestep, reward at the end is enough. Reinforcement learning can deal with temporal credit assignment problem, all algorithms are designed to work with ...
user avatar
  • 2,226
4 votes

In RL, if I assign the rewards for better positional play, the algorithm is learning nothing?

What you are proposing is closer to a heuristic for searching than a reward for RL. This is a blurred line, but generally if you start analysing the problem yourself, breaking it down into components ...
user avatar
  • 23.8k
4 votes
Accepted

Why is regret so defined in MABs?

In short, you don't regret your bad luck that you could do nothing about, you regret your bad choices that you could have done something about if only you knew. The point of regret as a metric ...
user avatar
  • 23.8k
4 votes
Accepted

Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?

I believe the claim is true. Here is my attempt at a proof. Let us consider the optimal infinite horizon value function $V_\alpha^*$ of $\mathcal{M}_\alpha$ at an arbitrary state $s \in S$. The value $...
user avatar
  • 381
3 votes
Accepted

How to evaluate an RL algorithm when used in a game?

When you want to compare Reinforcement Learning algorithms, you might want to compare the average rewards they generate and how fast and close they get to the optimal policy. However, in the case of ...
user avatar
  • 365
3 votes
Accepted

Are there any reliable ways of modifying the reward function to make the rewards less sparse?

Doing something like the dense, distance-based reward signal you propose is possible... but you have to do it very carefully. If you're not careful, and do it in a naive manner, you are likely to ...
user avatar
  • 9,379
3 votes

Is my interpretation of the return correct?

Your table is almost correct. It is a minor difference, you should not have a $R_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R_1$ (a result of ...
user avatar
  • 23.8k
3 votes
Accepted

How should I take into consideration the number of steps in the reward function?

How would you implement this "Number of Steps" cost? What the paper is referring to is the reward discounting process which is a standard way of formulating RL problems, either continuous ...
user avatar
  • 23.8k
3 votes

What would motivate a machine?

I asked professor Richard Sutton a similar question, in the first lecture of the reinforcement learning course. It seems that there are different ways to motivate the machine. In fact, machine ...
user avatar
  • 251
3 votes
Accepted

Do all expert trajectories have the same starting state in apprenticeship learning?

All right, I figured it out. trajectories need not have the same starting state because the distribution of $s_0$ is drawn from a distribution D (mentioned in the paper). Had been confused because ...
user avatar
  • 1,211
3 votes
Accepted

Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in ...
user avatar
  • 23.8k
3 votes
Accepted

Is there any difference between reward and return in reinforcement learning?

Return refers to the total discounted reward, starting from the current timestep.
user avatar
3 votes
Accepted

If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?

First note that $\mathbb{E}[R_{t+1} |S_t=s] = \sum_{s',r}rm(s',r|s)$ where $m(\cdot)$ is the mass function for the joint distribution of $S_{t+1},R_{t+1}$. If you are currently in state $S_t$ and we ...
user avatar
3 votes

Non-differentiable reward function to update a neural network

I cannot wrap my head around the concept of accuracy as a non-differentiable reward function. Do we need to find the function and then check if it is mathematically non-differentiable? In ...
user avatar
  • 23.8k
3 votes
Accepted

How is the reward in reinforcement learning different from the label in supervised learning problems?

Reward in reinforcement learning (RL) is entirely different from a supervised learning (SL) label, but can be related to it indirectly. In a RL control setting, you can imagine that you had a data ...
user avatar
  • 23.8k
3 votes
Accepted

How do we derive the expression for average reward setting in continuing tasks?

We assume that our MDP is ergodic. Loosely speaking, this means that wherever the MDP starts (i.e. no matter which state we start in) or any actions the agent takes early on can only have a limited ...
user avatar
3 votes

What is the difference between a reward and a value for a given state?

What is the difference between a reward and a value for a given state? Let us say that an agent took an action from state $A$ and reached state $B$ and got a score $R$. This instantaneous score the ...
user avatar
3 votes
Accepted

What is the difference between a reward and a value for a given state?

Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for ...
user avatar
  • 23.8k
3 votes

What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?

1. Question: The difference between loss and reward/penalty So I see both the loss function and the reward/penalty are the quantitative way of measuring the output/action and making the model to ...
user avatar
  • 346
3 votes

What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?

Ultimately, in RL, the policy is what you want to find. It's the solution to the Markov Decision Process (MDP). But you don't want to find any policy, but the optimal policy, i.e. the one that will ...
user avatar
  • 33.8k

Only top scored, non community-wiki answers of a minimum length are eligible