21
votes
Accepted
How should I handle invalid actions (when using REINFORCE)?
Just ignore the invalid moves.
For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If ...
- 4,185
14
votes
How should I handle invalid actions (when using REINFORCE)?
Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and ...
- 3,777
12
votes
Accepted
Can reinforcement learning be used for tasks where only one final reward is received?
RL can be used for cases where you have sparse rewards (i.e. at almost every step all rewards are zero), but, in such a setting, the experience the agent receives during the trajectory does not ...
- 39.1k
9
votes
Suitable reward function for trading buy and sell orders
Generally researchers (Ghandar et al, Michalewicz, Lam) have used the profit or return on investment (ROI) as a reward (fitness) function.
$ROI = \frac{ \left[\sum_{t=1}^T (Price_t - sc) \times I_s(t) ...
- 436
8
votes
Accepted
How could I use reinforcement learning to solve a chess-like board game?
I would like to use reinforcement learning to make the engine improve by playing against itself. I have been reading about the topic but I am still quite confused.
Be warned: Reinforcement learning ...
- 29k
7
votes
Accepted
What are some best practices when trying to design a reward function?
Designing reward functions
Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three ...
- 39.1k
7
votes
How should I handle invalid actions (when using REINFORCE)?
I faced a similar issue recently with Minesweeper.
The way I solved it was by ignoring the illegal/invalid moves entirely.
Use the Q-network to predict the Q-values for all of your actions (valid ...
- 153
6
votes
How should I handle invalid actions (when using REINFORCE)?
IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative ...
- 543
6
votes
What are some best practices when trying to design a reward function?
If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with ...
- 347
5
votes
Accepted
Can rewards be decomposed into components?
If I understood correctly you're looking at a Multi-Objective Reinforcement Learning (MORL). Keep in mind however that many scientist will often follow the reward hypothesis (Sutton and Barto) which ...
5
votes
Accepted
How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?
There seem to be two different ideas in this question here:
What's the impact / importance of our choice for reward values?
What's the impact / importance of our choice for initial value estimates (...
- 9,944
5
votes
Accepted
How do we define the reward function for an environment?
In Reinforcement Learning (RL), a reward function is part of the problem definition and should:
Be based primarily on the goals of the agent.
Take into account any combination of starting state $s$, ...
- 29k
4
votes
Counterexamples to the reward hypothesis
What if a scalar reward is insufficient, or its unclear on how to collapse a multi-dimensional reward to a single dimension. Example, for someone eating a burger, both taste and cost are important. ...
- 141
4
votes
Accepted
Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?
Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in ...
- 29k
4
votes
Can the rewards be stochastic when the transition model is deterministic?
My question is, would $r_1 =r_2$?
That's usually up to you as the designer of the system.
Usually when you declare that you have "a deterministic environment", you imply that both $s'$ and $r$ are ...
- 29k
4
votes
Accepted
Why doesn't Reward Normalization subtract the mean?
Reward normalisation is constrained by the need to not change the problem definition. The optimal policy should be the same with or without reward normalisation. The intended improvement is to make it ...
- 29k
3
votes
Accepted
How should I handle invalid actions in a grid world?
In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge.
It may help if you think through what the actual consequences ...
- 29k
3
votes
Accepted
Is a reward given at every step or only given when the RL agent fails or succeeds?
In reinforcement learning (RL), an immediate reward value must be returned after each action, along with the next state. This value can be zero though, which will have no direct impact on optimality ...
- 29k
3
votes
Accepted
Are there any reliable ways of modifying the reward function to make the rewards less sparse?
Doing something like the dense, distance-based reward signal you propose is possible... but you have to do it very carefully. If you're not careful, and do it in a naive manner, you are likely to ...
- 9,944
3
votes
Accepted
What should I do when the potential value of a state is too high?
Dennis Soemers provides an important point that from a theoretical standpoint, this can be seen as a non-issue. However, what you bring up is an important practical issue of potential-based reward ...
- 146
3
votes
What should I do when the potential value of a state is too high?
I don't think the situation you're sketching should be a problem at all.
If $P(s)$ is high (e.g. $P(s) = 1000$), this means (according to your shaping / "heuristic") that it's valuable to be in the ...
- 9,944
3
votes
Accepted
Reinforcement Learning algorithm with rewards dependent both on previous action and current action
The answer to both your concerns is:
Add the previous action choice to the state representation.
It is all you need to do. It gives the agent the data it needs to learn the association of negative ...
- 29k
3
votes
Accepted
Can the rewards be matrices when using DQN?
Generally speaking, is it better for rewards to be a scalar, or is using matrices okay?
Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and ...
- 29k
3
votes
Accepted
In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?
The main thing you will need to do is add the accumulated reward (total_score_so_far) to the state. In order to predict future reward with any accuracy, the agent ...
- 29k
3
votes
Accepted
Reward design or Inverse reinforcement learning?
It depends on the domain you are in.
Inverse RL (IRL) would be most advantageous in domains in which:
It's hard to specify the reward by hand: for example, it would be hard to hand-specify a reward ...
- 380
2
votes
How should I handle invalid actions (when using REINFORCE)?
An experimental paper exist in arxiv about the effect of whether to mask or to give negative rewards to invalid actions. There are some references in this paper which also discuss the effects and the ...
- 165
2
votes
Accepted
Expressing Arbitrary Reward Functions as Potential-Based Advice (PBA)
Is the method itself defective or anything wrong with my code?
There does indeed appear to be an issue with the code, the publications are fine (I know most of those authors and would very much trust ...
- 9,944
2
votes
Accepted
How should I define the reward function in the case of Connect Four?
The classic working reward scheme for two player zero sum games (i.e. if I win, you lose and vice versa) is simply:
+1 for a win
0 for a draw
-1 for a loss
These rewards should be associated with ...
- 29k
2
votes
Can rewards be decomposed into components?
I agree with Tomasz that the approach you are describing falls within the field of MORL. For a solid introduction MORL I would recommend the survey by Roijers, D. M., Vamplew, P., Whiteson, S., & ...
- 121
2
votes
Counterexamples to the reward hypothesis
I believe that there is no clear answer to your question. It essentially boils down to whether you are a reductionist – whether you believe that quantitative measurements can truly give justice to the ...
- 380
Only top scored, non community-wiki answers of a minimum length are eligible