Share Your Experience: Take the 2024 Developer Survey

# Tag Info

Accepted

### Why does the "reward to go" trick in policy gradient methods work?

An important thing we're going to need is what is called the "Expected Grad-Log-Prob Lemma here" (proof included on that page), which says that (for any $t$): \mathbb{E}_{\tau \sim \pi_{\...
• 10.3k
Accepted

### Why is the reward in reinforcement learning always a scalar?

If you have multiple types of rewards (say, R1 and R2), then it is no longer clear what would be the optimal way to act: it can happen that one way of acting would maximize R1 and another way would ...
• 241

### Why is the reward in reinforcement learning always a scalar?

Rather than the survey by Liu et al. recommended above, I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a ...
Accepted

### What is the difference between expected return and value function?

There is a strong relationship between a value function and a return. Namely that a value function calculates the expected return from being in a certain state, or taking a specific action in a ...
• 32.4k

### How do I handle negative rewards in policy gradients with the cross-entropy loss function?

It depends on your loss function, but you probably need to tweak it. If you are using an update rule like loss = -log(probabilities) * reward, then your loss is ...
• 171

### Why is the reward in reinforcement learning always a scalar?

Markov decision problems are usually defined with a reward function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and in these cases the rewards are expected to be scalar real values. This ...
• 562
Accepted

### What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?

1. Question: The difference between loss and reward/penalty So I see both the loss function and the reward/penalty are the quantitative way of measuring the output/action and making the model to ...
• 1,748
Accepted

### Is there any difference between reward and return in reinforcement learning?

Return refers to the total discounted reward, starting from the current timestep.
• 1,151
Accepted

### Why do my rewards reduce after extensive training using D3QN?

It is not 100% clear, but this seems like an instance of catastrophic forgetting. This is something that often impacts reinforcement learning. I have answered a very similar question on Data Science ...
• 32.4k

### What would motivate a machine?

This is an interesting question actually. There's a quite realistic idea about "where can the curiosity originate from" in the book "On intelligence" written by Jeff Hawkins and Sandra Blakeslee. It'...
• 109
Accepted

### What would motivate a machine?

The current method to implement motivation is some kind of artificial reward. Deepmind's DQN for example is driven by the score of the game. The higher the score, the better. The AI learns to adjust ...
• 1,684
Accepted

### Should RL rewards diminish over time?

RL agents - implemented correctly - do not take previous rewards into account when making decisions. For instance value functions only assess potential future reward. The state value or expected ...
• 32.4k

### Why cannot an AI agent adjust the reward function directly?

Why do both approaches prevent the AI agent from changing its reward function at will? In RL for optimal control, the reward function is part of the problem formulation. That is, it describes the ...
• 32.4k
Accepted

### If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?

First note that $\mathbb{E}[R_{t+1} |S_t=s] = \sum_{s',r}rm(s',r|s)$ where $m(\cdot)$ is the mass function for the joint distribution of $S_{t+1},R_{t+1}$. If you are currently in state $S_t$ and we ...
• 4,910

### Reinforcement Learning with long term rewards and fixed states and actions

You don't need to have a reward on every single timestep, reward at the end is enough. Reinforcement learning can deal with temporal credit assignment problem, all algorithms are designed to work with ...
• 2,406

### In RL, if I assign the rewards for better positional play, the algorithm is learning nothing?

What you are proposing is closer to a heuristic for searching than a reward for RL. This is a blurred line, but generally if you start analysing the problem yourself, breaking it down into components ...
• 32.4k

### Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity

(1) You might want look into RND (Random network distillation) which allows usage of a curiosity-based exploration bonus for the agent as an intrinsic reward. You can use the intrinsic reward to ...
• 171
Accepted

### Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in ...
• 32.4k

### Non-differentiable reward function to update a neural network

I cannot wrap my head around the concept of accuracy as a non-differentiable reward function. Do we need to find the function and then check if it is mathematically non-differentiable? In ...
• 32.4k
Accepted

### Why is regret so defined in MABs?

In short, you don't regret your bad luck that you could do nothing about, you regret your bad choices that you could have done something about if only you knew. The point of regret as a metric ...
• 32.4k