# Tag Info

Generally researchers (Ghandar et al, Michalewicz, Lam) have used the profit or return on investment (ROI) as a reward (fitness) function. $ROI = \frac{ \left[\sum_{t=1}^T (Price_t - sc) \times I_s(t) ... 8 votes Accepted ### Why is the reward in reinforcement learning always a scalar? If you have multiple types of rewards (say, R1 and R2), then it is no longer clear what would be the optimal way to act: it can happen that one way of acting would maximize R1 and another way would ... 7 votes ### How are the reward functions$R(s)$,$R(s, a)$and$R(s, a, s')$equivalent? In general the different reward functions$R(s)$,$R(s, a)$and$R(s, a, s')$are not equivalent mathematically, so you will not find any formal proof. It is possible for the functions to resolve to ... 7 votes ### Why is the reward in reinforcement learning always a scalar? Rather than the survey by Liu et al. recommended above, I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a ... 6 votes Accepted ### Why does the definition of the reward function$r(s, a, s')$involve the term$p(s' \mid s, a)$? Expectation of reward after taking action$a$in state$s$and ending up in state$s'$would simply be \begin{equation} r(s, a, s') = \sum_{r \in R} r \cdot p(r|s, a, s') \end{equation} The problem ... 6 votes ### Why is the reward in reinforcement learning always a scalar? Markov decision problems are usually defined with a reward function$r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and in these cases the rewards are expected to be scalar real values. This ... 5 votes Accepted ### Can rewards be decomposed into components? If I understood correctly you're looking at a Multi-Objective Reinforcement Learning (MORL). Keep in mind however that many scientist will often follow the reward hypothesis (Sutton and Barto) which ... 5 votes Accepted ### What are some best practices when trying to design a reward function? Designing reward functions Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three ... 5 votes Accepted ### How can I ensure convergence of DDQN, if the true Q-values for different actions in the same state are very close? Let$Q^*(s, a)$denote the "true"$Q$-value for a state-action pair$(s, a)$, i.e. the values that we're hoping to learn to approximate using a neural network that outputs$Q(s, a)$values. The ... 5 votes Accepted ### What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative? 1) Is there any way to set the initial Q-values for the actions? You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network ... 5 votes Accepted ### How does the initialization of the value function and definition of the reward function affect the performance of the RL agent? There seem to be two different ideas in this question here: What's the impact / importance of our choice for reward values? What's the impact / importance of our choice for initial value estimates (... 4 votes ### Counterexamples to the reward hypothesis What if a scalar reward is insufficient, or its unclear on how to collapse a multi-dimensional reward to a single dimension. Example, for someone eating a burger, both taste and cost are important. ... 4 votes Accepted ### How do we define the reward function for an environment? In Reinforcement Learning (RL), a reward function is part of the problem definition and should: Be based primarily on the goals of the agent. Take into account any combination of starting state$s$, ... 4 votes Accepted ### Why is the equation$r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true? No, the substitution you suggest based on Equation (3.4) is not correct because you forgot about the$\sum_{r \in \mathcal{R}}$in the right-hand side Equation (3.4). Equation (3.4) says (leaving out ... 4 votes ### Can the rewards be stochastic when the transition model is deterministic? My question is, would$r_1 =r_2$? That's usually up to you as the designer of the system. Usually when you declare that you have "a deterministic environment", you imply that both$s'$and$r$are ... 4 votes ### How are the reward functions$R(s)$,$R(s, a)$and$R(s, a, s')$equivalent? Let$R(s)$denote a probability distribution over rewards that our agent may get in some MDP as a reward for entering a state$s$. The easiest case is to demonstrate that we can also choose to write ... 4 votes ### What are some best practices when trying to design a reward function? If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with ... 3 votes Accepted ### How do I convert an MDP with the reward function in the form$R(s,a,s')$to and an MDP with a reward function in the form$R(s,a)$? I think I may be in position to answer my own question. The Bellman equation (for the optimal policy) for a MDP with$r(s,a,s')$rewards would look like this: $$V(s) = \max_a \left\{ \sum_{s'} p(s'|s,... 3 votes Accepted ### How should I handle invalid actions in a grid world? In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge. It may help if you think through what the actual consequences ... 3 votes Accepted ### Why is the reward function \text{reward} = 1/{(\text{cost}+1)^2} better than \text{reward} =1/(\text{cost}+1)? Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in ... 3 votes Accepted ### Is a reward given at every step or only given when the RL agent fails or succeeds? In reinforcement learning (RL), an immediate reward value must be returned after each action, along with the next state. This value can be zero though, which will have no direct impact on optimality ... 3 votes Accepted ### What are the pros and cons of sparse and dense rewards in reinforcement learning? What are the pros and cons of sparse and dense rewards in reinforcement learning? It is unusual to refer to this difference as "pros and cons" because that term is often used to make ... 3 votes Accepted ### Can the rewards be matrices when using DQN? Generally speaking, is it better for rewards to be a scalar, or is using matrices okay? Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and ... 3 votes Accepted ### How to improve the reward signal when the rewards are sparse? Andrew Y. Ng (yes, that famous guy!) et al. proved, in the seminal paper Policy invariance under reward transformations: Theory and application to reward shaping (ICML, 1999), which was then part of ... 3 votes Accepted ### Intuition behind 1-\gamma and \frac{1}{1-\gamma} for calculating discounted future state distribution and discounted reward Question 1 The taylor expansion of \frac{1}{1-\gamma} at \gamma= 0 is as follows$$\frac{1}{1-\gamma} = 1 + \gamma + \gamma^2 + \dots$$When you multiply by 1-\gamma you get$$ 1 = (1-\gamma)(1 +... 3 votes ### How to deal with small reward values The numbers that a value-based neural network will predict are usually based on expected returns (sum of rewards by end of an episode, or a discounted infinite sum), although in some cases they might ... 2 votes Accepted ### Why does potential-based reward shaping seem to alter the optimal policy in this case? The same$\gamma = 0.9$that you use in the definition$F \doteq \gamma \Phi(s') - \Phi(s)$should also be used as the discount factor in computing returns for multi-step trajectories. So, rather than ... 2 votes Accepted ### Where are the parentheses in the definition of$r(s,a)\$?

Your first option is correct: $$r(s,a) = \mathbb{E}\left[R_t|S_{t-1}=s,A_{t-1}=a\right]=\sum_{r\in \mathcal{R}}\left[r\sum_{s'\in \mathcal{S}}p(s',r|s,a)\right]$$ It's partly a matter of taste, but ...