9 votes

Suitable reward function for trading buy and sell orders

Generally researchers (Ghandar et al, Michalewicz, Lam) have used the profit or return on investment (ROI) as a reward (fitness) function. $ROI = \frac{ \left[\sum_{t=1}^T (Price_t - sc) \times I_s(t) ...
user avatar
  • 436
8 votes
Accepted

Why is the reward in reinforcement learning always a scalar?

If you have multiple types of rewards (say, R1 and R2), then it is no longer clear what would be the optimal way to act: it can happen that one way of acting would maximize R1 and another way would ...
user avatar
  • 231
7 votes

How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?

In general the different reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ are not equivalent mathematically, so you will not find any formal proof. It is possible for the functions to resolve to ...
user avatar
  • 24.5k
7 votes

Why is the reward in reinforcement learning always a scalar?

Rather than the survey by Liu et al. recommended above, I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a ...
user avatar
6 votes
Accepted

Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?

Expectation of reward after taking action $a$ in state $s$ and ending up in state $s'$ would simply be \begin{equation} r(s, a, s') = \sum_{r \in R} r \cdot p(r|s, a, s') \end{equation} The problem ...
user avatar
  • 2,236
6 votes

Why is the reward in reinforcement learning always a scalar?

Markov decision problems are usually defined with a reward function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and in these cases the rewards are expected to be scalar real values. This ...
user avatar
5 votes
Accepted

Can rewards be decomposed into components?

If I understood correctly you're looking at a Multi-Objective Reinforcement Learning (MORL). Keep in mind however that many scientist will often follow the reward hypothesis (Sutton and Barto) which ...
user avatar
5 votes
Accepted

What are some best practices when trying to design a reward function?

Designing reward functions Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three ...
user avatar
  • 34.3k
5 votes
Accepted

How can I ensure convergence of DDQN, if the true Q-values for different actions in the same state are very close?

Let $Q^*(s, a)$ denote the "true" $Q$-value for a state-action pair $(s, a)$, i.e. the values that we're hoping to learn to approximate using a neural network that outputs $Q(s, a)$ values. The ...
user avatar
  • 9,419
5 votes
Accepted

What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?

1) Is there any way to set the initial Q-values for the actions? You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network ...
user avatar
5 votes
Accepted

How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?

There seem to be two different ideas in this question here: What's the impact / importance of our choice for reward values? What's the impact / importance of our choice for initial value estimates (...
user avatar
  • 9,419
4 votes

Counterexamples to the reward hypothesis

What if a scalar reward is insufficient, or its unclear on how to collapse a multi-dimensional reward to a single dimension. Example, for someone eating a burger, both taste and cost are important. ...
user avatar
  • 141
4 votes
Accepted

How do we define the reward function for an environment?

In Reinforcement Learning (RL), a reward function is part of the problem definition and should: Be based primarily on the goals of the agent. Take into account any combination of starting state $s$, ...
user avatar
  • 24.5k
4 votes
Accepted

Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true?

No, the substitution you suggest based on Equation (3.4) is not correct because you forgot about the $\sum_{r \in \mathcal{R}}$ in the right-hand side Equation (3.4). Equation (3.4) says (leaving out ...
user avatar
  • 9,419
4 votes

Can the rewards be stochastic when the transition model is deterministic?

My question is, would $r_1 =r_2$? That's usually up to you as the designer of the system. Usually when you declare that you have "a deterministic environment", you imply that both $s'$ and $r$ are ...
user avatar
  • 24.5k
4 votes

How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?

Let $R(s)$ denote a probability distribution over rewards that our agent may get in some MDP as a reward for entering a state $s$. The easiest case is to demonstrate that we can also choose to write ...
user avatar
  • 9,419
4 votes

What are some best practices when trying to design a reward function?

If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with ...
user avatar
  • 317
3 votes
Accepted

How do I convert an MDP with the reward function in the form $R(s,a,s')$ to and an MDP with a reward function in the form $R(s,a)$?

I think I may be in position to answer my own question. The Bellman equation (for the optimal policy) for a MDP with $r(s,a,s')$ rewards would look like this: $$V(s) = \max_a \left\{ \sum_{s'} p(s'|s,...
user avatar
  • 436
3 votes
Accepted

How should I handle invalid actions in a grid world?

In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge. It may help if you think through what the actual consequences ...
user avatar
  • 24.5k
3 votes
Accepted

Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in ...
user avatar
  • 24.5k
3 votes
Accepted

Is a reward given at every step or only given when the RL agent fails or succeeds?

In reinforcement learning (RL), an immediate reward value must be returned after each action, along with the next state. This value can be zero though, which will have no direct impact on optimality ...
user avatar
  • 24.5k
3 votes
Accepted

What are the pros and cons of sparse and dense rewards in reinforcement learning?

What are the pros and cons of sparse and dense rewards in reinforcement learning? It is unusual to refer to this difference as "pros and cons" because that term is often used to make ...
user avatar
  • 24.5k
3 votes
Accepted

Can the rewards be matrices when using DQN?

Generally speaking, is it better for rewards to be a scalar, or is using matrices okay? Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and ...
user avatar
  • 24.5k
3 votes
Accepted

How to improve the reward signal when the rewards are sparse?

Andrew Y. Ng (yes, that famous guy!) et al. proved, in the seminal paper Policy invariance under reward transformations: Theory and application to reward shaping (ICML, 1999), which was then part of ...
user avatar
  • 34.3k
3 votes
Accepted

Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward

Question 1 The taylor expansion of $\frac{1}{1-\gamma}$ at $\gamma= 0$ is as follows $$\frac{1}{1-\gamma} = 1 + \gamma + \gamma^2 + \dots$$ When you multiply by $1-\gamma$ you get $$ 1 = (1-\gamma)(1 +...
user avatar
3 votes

How to deal with small reward values

The numbers that a value-based neural network will predict are usually based on expected returns (sum of rewards by end of an episode, or a discounted infinite sum), although in some cases they might ...
user avatar
  • 24.5k
2 votes
Accepted

Why does potential-based reward shaping seem to alter the optimal policy in this case?

The same $\gamma = 0.9$ that you use in the definition $F \doteq \gamma \Phi(s') - \Phi(s)$ should also be used as the discount factor in computing returns for multi-step trajectories. So, rather than ...
user avatar
  • 9,419
2 votes
Accepted

Where are the parentheses in the definition of $r(s,a)$?

Your first option is correct: $$r(s,a) = \mathbb{E}\left[R_t|S_{t-1}=s,A_{t-1}=a\right]=\sum_{r\in \mathcal{R}}\left[r\sum_{s'\in \mathcal{S}}p(s',r|s,a)\right]$$ It's partly a matter of taste, but ...
user avatar
  • 24.5k
2 votes
Accepted

What is the reward system of reinforcement learning?

In this case, the word "system" refers to a Markov decision process (MDP), which is the mathematical model used to represent the reinforcement learning (RL) problem or, in general, a decision making ...
user avatar
  • 34.3k
2 votes

Is the policy really invariant under affine transformations of the reward function?

This statement: (it is well-known that the optimal policy is invariant to positive affine transformation of the reward function). is, as far as I know, and as you summarise, incorrect* because ...
user avatar
  • 24.5k

Only top scored, non community-wiki answers of a minimum length are eligible