18 votes
Accepted

How should I handle invalid actions (when using REINFORCE)?

Just ignore the invalid moves. For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If ...
user avatar
14 votes

How should I handle invalid actions (when using REINFORCE)?

Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and ...
user avatar
10 votes
Accepted

Can reinforcement learning be used for tasks where only one final reward is received?

RL can be used for cases where you have sparse rewards (i.e. at almost every step all rewards are zero), but, in such a setting, the experience the agent receives during the trajectory does not ...
user avatar
  • 33k
9 votes

Suitable reward function for trading buy and sell orders

Generally researchers (Ghandar et al, Michalewicz, Lam) have used the profit or return on investment (ROI) as a reward (fitness) function. $ROI = \frac{ \left[\sum_{t=1}^T (Price_t - sc) \times I_s(t) ...
user avatar
  • 436
7 votes

How should I handle invalid actions (when using REINFORCE)?

I faced a similar issue recently with Minesweeper. The way I solved it was by ignoring the illegal/invalid moves entirely. Use the Q-network to predict the Q-values for all of your actions (valid ...
user avatar
  • 133
6 votes

How should I handle invalid actions (when using REINFORCE)?

IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative ...
user avatar
6 votes
Accepted

How could I use reinforcement learning to solve a chess-like board game?

I would like to use reinforcement learning to make the engine improve by playing against itself. I have been reading about the topic but I am still quite confused. Be warned: Reinforcement learning ...
user avatar
  • 23.1k
5 votes
Accepted

Can rewards be decomposed into components?

If I understood correctly you're looking at a Multi-Objective Reinforcement Learning (MORL). Keep in mind however that many scientist will often follow the reward hypothesis (Sutton and Barto) which ...
user avatar
5 votes
Accepted

What are some best practices when trying to design a reward function?

Designing reward functions Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three ...
user avatar
  • 33k
5 votes
Accepted

How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?

There seem to be two different ideas in this question here: What's the impact / importance of our choice for reward values? What's the impact / importance of our choice for initial value estimates (...
user avatar
  • 9,316
4 votes
Accepted

How do we define the reward function for an environment?

In Reinforcement Learning (RL), a reward function is part of the problem definition and should: Be based primarily on the goals of the agent. Take into account any combination of starting state $s$, ...
user avatar
  • 23.1k
4 votes

Counterexamples to the reward hypothesis

What if a scalar reward is insufficient, or its unclear on how to collapse a multi-dimensional reward to a single dimension. Example, for someone eating a burger, both taste and cost are important. ...
user avatar
  • 141
4 votes

What are some best practices when trying to design a reward function?

If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with ...
user avatar
  • 337
4 votes

Can the rewards be stochastic when the transition model is deterministic?

My question is, would $r_1 =r_2$? That's usually up to you as the designer of the system. Usually when you declare that you have "a deterministic environment", you imply that both $s'$ and $r$ are ...
user avatar
  • 23.1k
3 votes
Accepted

Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in ...
user avatar
  • 23.1k
3 votes
Accepted

Are there any reliable ways of modifying the reward function to make the rewards less sparse?

Doing something like the dense, distance-based reward signal you propose is possible... but you have to do it very carefully. If you're not careful, and do it in a naive manner, you are likely to ...
user avatar
  • 9,316
3 votes
Accepted

What should I do when the potential value of a state is too high?

Dennis Soemers provides an important point that from a theoretical standpoint, this can be seen as a non-issue. However, what you bring up is an important practical issue of potential-based reward ...
user avatar
3 votes

What should I do when the potential value of a state is too high?

I don't think the situation you're sketching should be a problem at all. If $P(s)$ is high (e.g. $P(s) = 1000$), this means (according to your shaping / "heuristic") that it's valuable to be in the ...
user avatar
  • 9,316
3 votes
Accepted

Can the rewards be matrices when using DQN?

Generally speaking, is it better for rewards to be a scalar, or is using matrices okay? Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and ...
user avatar
  • 23.1k
2 votes

How should I handle invalid actions (when using REINFORCE)?

An experimental paper exist in arxiv about the effect of whether to mask or to give negative rewards to invalid actions. There are some references in this paper which also discuss the effects and the ...
user avatar
  • 143
2 votes
Accepted

How should I define the reward function in the case of Connect Four?

The classic working reward scheme for two player zero sum games (i.e. if I win, you lose and vice versa) is simply: +1 for a win 0 for a draw -1 for a loss These rewards should be associated with ...
user avatar
  • 23.1k
2 votes
Accepted

Is a reward given at every step or only given when the RL agent fails or succeeds?

In reinforcement learning (RL), an immediate reward value must be returned after each action, along with the next state. This value can be zero though, which will have no direct impact on optimality ...
user avatar
  • 23.1k
2 votes

Can rewards be decomposed into components?

I agree with Tomasz that the approach you are describing falls within the field of MORL. For a solid introduction MORL I would recommend the survey by Roijers, D. M., Vamplew, P., Whiteson, S., & ...
user avatar
2 votes
Accepted

How should I handle invalid actions in a grid world?

In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge. It may help if you think through what the actual consequences ...
user avatar
  • 23.1k
2 votes

How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?

Sutton and Barto state, "The reward signal is your way of communicating to the robot [agent] what you want it to achieve, not how you want it achieved." Since you stated that the goal is to reach the ...
user avatar
  • 776
2 votes

Counterexamples to the reward hypothesis

I believe that there is no clear answer to your question. It essentially boils down to whether you are a reductionist – whether you believe that quantitative measurements can truly give justice to the ...
user avatar
  • 101
2 votes
Accepted

Reinforcement Learning algorithm with rewards dependent both on previous action and current action

The answer to both your concerns is: Add the previous action choice to the state representation. It is all you need to do. It gives the agent the data it needs to learn the association of negative ...
user avatar
  • 23.1k
2 votes

How to construct a reward function for a "wait and see" problem

In general, the term of art for this problem is "early classification." Early classification of time series has been extensively studied for minimizing class prediction delay in time-...
user avatar
  • 236
1 vote

How should I define the reward function to solve the Wumpus game with deep Q-learning?

The reward function belongs the the environment and it is the only way the agent can explore the world given a state. If we want agent to do something specific, we must provide rewards to it in such a ...
user avatar
1 vote

How should I define the reward function to solve the Wumpus game with deep Q-learning?

The reward function is up to you when you set the goals for the agent. If the goal is to score as highly as possible, before ending the game, then use the score. You may want to scale the score down ...
user avatar
  • 23.1k

Only top scored, non community-wiki answers of a minimum length are eligible