Skip to main content
Share Your Experience: Take the 2024 Developer Survey
9 votes

Suitable reward function for trading buy and sell orders

Generally researchers (Ghandar et al, Michalewicz, Lam) have used the profit or return on investment (ROI) as a reward (fitness) function. $ROI = \frac{ \left[\sum_{t=1}^T (Price_t - sc) \times I_s(t) ...
Jason's user avatar
  • 436
9 votes
Accepted

What are some best practices when trying to design a reward function?

Designing reward functions Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three ...
nbro's user avatar
  • 40.8k
9 votes
Accepted

Why is the reward in reinforcement learning always a scalar?

If you have multiple types of rewards (say, R1 and R2), then it is no longer clear what would be the optimal way to act: it can happen that one way of acting would maximize R1 and another way would ...
present's user avatar
  • 241
8 votes

How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?

In general the different reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ are not equivalent mathematically, so you will not find any formal proof. It is possible for the functions to resolve to ...
Neil Slater's user avatar
  • 32.5k
8 votes
Accepted

Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?

Expectation of reward after taking action $a$ in state $s$ and ending up in state $s'$ would simply be \begin{equation} r(s, a, s') = \sum_{r \in R} r \cdot p(r|s, a, s') \end{equation} The problem ...
Brale's user avatar
  • 2,406
8 votes

Why is the reward in reinforcement learning always a scalar?

Rather than the survey by Liu et al. recommended above, I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a ...
Peter Vamplew's user avatar
7 votes

Why is the reward in reinforcement learning always a scalar?

Markov decision problems are usually defined with a reward function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and in these cases the rewards are expected to be scalar real values. This ...
Hai Nguyen's user avatar
6 votes

What are some best practices when trying to design a reward function?

If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with ...
user76284's user avatar
  • 347
5 votes
Accepted

Can rewards be decomposed into components?

If I understood correctly you're looking at a Multi-Objective Reinforcement Learning (MORL). Keep in mind however that many scientist will often follow the reward hypothesis (Sutton and Barto) which ...
Tomasz Bartkowiak's user avatar
5 votes
Accepted

How can I ensure convergence of DDQN, if the true Q-values for different actions in the same state are very close?

Let $Q^*(s, a)$ denote the "true" $Q$-value for a state-action pair $(s, a)$, i.e. the values that we're hoping to learn to approximate using a neural network that outputs $Q(s, a)$ values. The ...
Dennis Soemers's user avatar
  • 10.3k
5 votes
Accepted

Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true?

No, the substitution you suggest based on Equation (3.4) is not correct because you forgot about the $\sum_{r \in \mathcal{R}}$ in the right-hand side Equation (3.4). Equation (3.4) says (leaving out ...
Dennis Soemers's user avatar
  • 10.3k
5 votes

How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?

Let $R(s)$ denote a probability distribution over rewards that our agent may get in some MDP as a reward for entering a state $s$. The easiest case is to demonstrate that we can also choose to write ...
Dennis Soemers's user avatar
  • 10.3k
5 votes
Accepted

What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?

1) Is there any way to set the initial Q-values for the actions? You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network ...
BlueMoon93's user avatar
5 votes
Accepted

How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?

There seem to be two different ideas in this question here: What's the impact / importance of our choice for reward values? What's the impact / importance of our choice for initial value estimates (...
Dennis Soemers's user avatar
  • 10.3k
5 votes
Accepted

How do we define the reward function for an environment?

In Reinforcement Learning (RL), a reward function is part of the problem definition and should: Be based primarily on the goals of the agent. Take into account any combination of starting state $s$, ...
Neil Slater's user avatar
  • 32.5k
5 votes
Accepted

How to improve the reward signal when the rewards are sparse?

Andrew Y. Ng (yes, that famous guy!) et al. proved, in the seminal paper Policy invariance under reward transformations: Theory and application to reward shaping (ICML, 1999), which was then part of ...
nbro's user avatar
  • 40.8k
4 votes

Counterexamples to the reward hypothesis

What if a scalar reward is insufficient, or its unclear on how to collapse a multi-dimensional reward to a single dimension. Example, for someone eating a burger, both taste and cost are important. ...
Kumar's user avatar
  • 141
4 votes

Can the rewards be stochastic when the transition model is deterministic?

My question is, would $r_1 =r_2$? That's usually up to you as the designer of the system. Usually when you declare that you have "a deterministic environment", you imply that both $s'$ and $r$ are ...
Neil Slater's user avatar
  • 32.5k
4 votes
Accepted

Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in ...
Neil Slater's user avatar
  • 32.5k
4 votes
Accepted

What are the pros and cons of sparse and dense rewards in reinforcement learning?

What are the pros and cons of sparse and dense rewards in reinforcement learning? It is unusual to refer to this difference as "pros and cons" because that term is often used to make ...
Neil Slater's user avatar
  • 32.5k
4 votes
Accepted

How to deal with small reward values

The numbers that a value-based neural network will predict are usually based on expected returns (sum of rewards by end of an episode, or a discounted infinite sum), although in some cases they might ...
Neil Slater's user avatar
  • 32.5k
3 votes

Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?

$\frac{p(s', r \mid s, a)}{p(s' \mid s, a)}$ represents the probability of observing reward $r$ in state $s'$, given that state $s'$ is the next state transitioned to. The equation assumes a ...
calveeen's user avatar
  • 1,271
3 votes
Accepted

How do I convert an MDP with the reward function in the form $R(s,a,s')$ to and an MDP with a reward function in the form $R(s,a)$?

I think I may be in position to answer my own question. The Bellman equation (for the optimal policy) for a MDP with $r(s,a,s')$ rewards would look like this: $$V(s) = \max_a \left\{ \sum_{s'} p(s'|s,...
Asher's user avatar
  • 436
3 votes
Accepted

How should I handle invalid actions in a grid world?

In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge. It may help if you think through what the actual consequences ...
Neil Slater's user avatar
  • 32.5k
3 votes
Accepted

Is a reward given at every step or only given when the RL agent fails or succeeds?

In reinforcement learning (RL), an immediate reward value must be returned after each action, along with the next state. This value can be zero though, which will have no direct impact on optimality ...
Neil Slater's user avatar
  • 32.5k
3 votes
Accepted

How to apply Q-learning when rewards is only available at the last state?

Having only a non-zero reward at the very end is not uncommon. When rewards are sparse, it becomes a bit harder to learn compared to having lots of different rewards along the way, but for your ...
Robby Goetschalckx's user avatar
3 votes
Accepted

Can the rewards be matrices when using DQN?

Generally speaking, is it better for rewards to be a scalar, or is using matrices okay? Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and ...
Neil Slater's user avatar
  • 32.5k
3 votes
Accepted

Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward

Question 1 The taylor expansion of $\frac{1}{1-\gamma}$ at $\gamma= 0$ is as follows $$\frac{1}{1-\gamma} = 1 + \gamma + \gamma^2 + \dots$$ When you multiply by $1-\gamma$ you get $$ 1 = (1-\gamma)(1 +...
quest ions's user avatar
3 votes
Accepted

In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?

The main thing you will need to do is add the accumulated reward (total_score_so_far) to the state. In order to predict future reward with any accuracy, the agent ...
Neil Slater's user avatar
  • 32.5k
3 votes
Accepted

Reward design or Inverse reinforcement learning?

It depends on the domain you are in. Inverse RL (IRL) would be most advantageous in domains in which: It's hard to specify the reward by hand: for example, it would be hard to hand-specify a reward ...
mdc's user avatar
  • 380

Only top scored, non community-wiki answers of a minimum length are eligible