9
votes
Suitable reward function for trading buy and sell orders
Generally researchers (Ghandar et al, Michalewicz, Lam) have used the profit or return on investment (ROI) as a reward (fitness) function.
$ROI = \frac{ \left[\sum_{t=1}^T (Price_t - sc) \times I_s(t) ...
- 436
8
votes
Accepted
Why is the reward in reinforcement learning always a scalar?
If you have multiple types of rewards (say, R1 and R2), then it is no longer clear what would be the optimal way to act: it can happen that one way of acting would maximize R1 and another way would ...
- 231
7
votes
How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?
In general the different reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ are not equivalent mathematically, so you will not find any formal proof.
It is possible for the functions to resolve to ...
- 30.2k
7
votes
Accepted
What are some best practices when trying to design a reward function?
Designing reward functions
Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three ...
- 39.5k
7
votes
Why is the reward in reinforcement learning always a scalar?
Rather than the survey by Liu et al. recommended above, I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a ...
- 121
6
votes
Accepted
Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?
Expectation of reward after taking action $a$ in state $s$ and ending up in state $s'$ would simply be
\begin{equation}
r(s, a, s') = \sum_{r \in R} r \cdot p(r|s, a, s')
\end{equation}
The problem ...
- 2,336
6
votes
What are some best practices when trying to design a reward function?
If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with ...
- 347
6
votes
Why is the reward in reinforcement learning always a scalar?
Markov decision problems are usually defined with a reward function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and in these cases the rewards are expected to be scalar real values. This ...
- 552
5
votes
Accepted
Can rewards be decomposed into components?
If I understood correctly you're looking at a Multi-Objective Reinforcement Learning (MORL). Keep in mind however that many scientist will often follow the reward hypothesis (Sutton and Barto) which ...
5
votes
How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?
Let $R(s)$ denote a probability distribution over rewards that our agent may get in some MDP as a reward for entering a state $s$. The easiest case is to demonstrate that we can also choose to write ...
- 10.1k
5
votes
Accepted
What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?
1) Is there any way to set the initial Q-values for the actions?
You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network ...
- 906
5
votes
Accepted
How can I ensure convergence of DDQN, if the true Q-values for different actions in the same state are very close?
Let $Q^*(s, a)$ denote the "true" $Q$-value for a state-action pair $(s, a)$, i.e. the values that we're hoping to learn to approximate using a neural network that outputs $Q(s, a)$ values.
The ...
- 10.1k
5
votes
Accepted
How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?
There seem to be two different ideas in this question here:
What's the impact / importance of our choice for reward values?
What's the impact / importance of our choice for initial value estimates (...
- 10.1k
5
votes
Accepted
How do we define the reward function for an environment?
In Reinforcement Learning (RL), a reward function is part of the problem definition and should:
Be based primarily on the goals of the agent.
Take into account any combination of starting state $s$, ...
- 30.2k
5
votes
Accepted
How to improve the reward signal when the rewards are sparse?
Andrew Y. Ng (yes, that famous guy!) et al. proved, in the seminal paper Policy invariance under reward transformations: Theory and application to reward shaping (ICML, 1999), which was then part of ...
- 39.5k
4
votes
Counterexamples to the reward hypothesis
What if a scalar reward is insufficient, or its unclear on how to collapse a multi-dimensional reward to a single dimension. Example, for someone eating a burger, both taste and cost are important. ...
- 141
4
votes
Can the rewards be stochastic when the transition model is deterministic?
My question is, would $r_1 =r_2$?
That's usually up to you as the designer of the system.
Usually when you declare that you have "a deterministic environment", you imply that both $s'$ and $r$ are ...
- 30.2k
4
votes
Accepted
Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true?
No, the substitution you suggest based on Equation (3.4) is not correct because you forgot about the $\sum_{r \in \mathcal{R}}$ in the right-hand side Equation (3.4).
Equation (3.4) says (leaving out ...
- 10.1k
4
votes
Accepted
Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?
Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in ...
- 30.2k
4
votes
Accepted
What are the pros and cons of sparse and dense rewards in reinforcement learning?
What are the pros and cons of sparse and dense rewards in reinforcement learning?
It is unusual to refer to this difference as "pros and cons" because that term is often used to make ...
- 30.2k
4
votes
Accepted
How to deal with small reward values
The numbers that a value-based neural network will predict are usually based on expected returns (sum of rewards by end of an episode, or a discounted infinite sum), although in some cases they might ...
- 30.2k
3
votes
Accepted
Is a reward given at every step or only given when the RL agent fails or succeeds?
In reinforcement learning (RL), an immediate reward value must be returned after each action, along with the next state. This value can be zero though, which will have no direct impact on optimality ...
- 30.2k
3
votes
Accepted
How do I convert an MDP with the reward function in the form $R(s,a,s')$ to and an MDP with a reward function in the form $R(s,a)$?
I think I may be in position to answer my own question. The Bellman equation (for the optimal policy) for a MDP with $r(s,a,s')$ rewards would look like this:
$$V(s) = \max_a \left\{ \sum_{s'} p(s'|s,...
- 436
3
votes
Accepted
How should I handle invalid actions in a grid world?
In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge.
It may help if you think through what the actual consequences ...
- 30.2k
3
votes
Accepted
How to apply Q-learning when rewards is only available at the last state?
Having only a non-zero reward at the very end is not uncommon. When rewards are sparse, it becomes a bit harder to learn compared to having lots of different rewards along the way, but for your ...
3
votes
Accepted
Can the rewards be matrices when using DQN?
Generally speaking, is it better for rewards to be a scalar, or is using matrices okay?
Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and ...
- 30.2k
3
votes
Accepted
Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward
Question 1
The taylor expansion of $\frac{1}{1-\gamma}$ at $\gamma= 0$ is as follows
$$\frac{1}{1-\gamma} = 1 + \gamma + \gamma^2 + \dots$$
When you multiply by $1-\gamma$ you get
$$ 1 = (1-\gamma)(1 +...
- 384
3
votes
Accepted
In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?
The main thing you will need to do is add the accumulated reward (total_score_so_far) to the state. In order to predict future reward with any accuracy, the agent ...
- 30.2k
3
votes
Accepted
Reward design or Inverse reinforcement learning?
It depends on the domain you are in.
Inverse RL (IRL) would be most advantageous in domains in which:
It's hard to specify the reward by hand: for example, it would be hard to hand-specify a reward ...
- 380
2
votes
Accepted
Why does potential-based reward shaping seem to alter the optimal policy in this case?
The same $\gamma = 0.9$ that you use in the definition $F \doteq \gamma \Phi(s') - \Phi(s)$ should also be used as the discount factor in computing returns for multi-step trajectories. So, rather than ...
- 10.1k
Only top scored, non community-wiki answers of a minimum length are eligible