People who code: we want your input. Take the Survey
14

Just ignore the invalid moves. For exploration it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If you only punish illegal moves they will still retain some probability (however small) and therefore will be executed from time to time (however seldom). So you ...


12

Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and the features $\phi$ is a function of the current state $s$ and an action from the set of actions $A$. $$ \pi(\theta, a) = \frac{e^{\theta \phi(s, a)}}{\sum_{b \...


10

RL can be used for cases where you have sparse rewards (i.e. at almost every step all rewards are zero), but, in such a setting, the experience the agent receives during the trajectory does not provide much information regarding the quality of the actions. Games can be often formulated as episodic tasks. For example, you could formulate a chess match as an ...


7

I faced a similar issue recently with Minesweeper. The way I solved it was by ignoring the illegal/invalid moves entirely. Use the Q-network to predict the Q-values for all of your actions (valid and invalid) Pre-process the Q-values by setting all of the invalid moves to a Q-value of zero/negative number (depends on your scenario) Use a policy of your ...


7

IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative reward. Absurd? Sure! But in fact your invalid moves are just a relic of the representation (which itself is straightforward and fine). The best treatment of them is to exclude them completely ...


6

I would like to use reinforcement learning to make the engine improve by playing against itself. I have been reading about the topic but I am still quite confused. Be warned: Reinforcement learning is a large complex subject. Although it might take you on a detour from game-playing bots, you may want to study RL basics. A good place to start is Sutton &...


5

If I understood correctly you're looking at a Multi-Objective Reinforcement Learning (MORL). Keep in mind however that many scientist will often follow the reward hypothesis (Sutton and Barto) which says that All of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar ...


4

If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with this reward function is that it's too sparse, meaning the agent has little guidance on how to modify their behavior to become better at attaining said goal, ...


4

Designing reward functions Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three outcomes: win (good), loss (bad), or draw (neutral). So, you could reward the agent with $+1$ if it wins the game, $-1$ if it loses, and $0$ if it draws (or for any ...


4

My question is, would $r_1 =r_2$? That's usually up to you as the designer of the system. Usually when you declare that you have "a deterministic environment", you imply that both $s'$ and $r$ are fixed values depending on $(s,a)$. So in your examples, you would expect your observations to also have $r_1 = r_2$ However, it is possible to define a MDP ...


4

There seem to be two different ideas in this question here: What's the impact / importance of our choice for reward values? What's the impact / importance of our choice for initial value estimates (how do we initialise our table of $Q(s, a)$ values in the case of a simple, tabular RL algorithm like Sarsa or $Q$-learning)? The reward values are typically ...


3

What if a scalar reward is insufficient, or its unclear on how to collapse a multi-dimensional reward to a single dimension. Example, for someone eating a burger, both taste and cost are important. Agents may prioritize taste and cost differently, so its not clear on how to aggregate the two. It is also not clear on how a subjective categorical taste value ...


3

Doing something like the dense, distance-based reward signal you propose is possible... but you have to do it very carefully. If you're not careful, and do it in a naive manner, you are likely to reinforce unwanted behaviour. For example, the way I read that reward function you propose, it provide a positive reward for any steps taken by the agent, with ...


3

Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in some cases change the priority of solutions. As a simple example, consider a choice between trajectories with costs A(0,4,4,4) and B(1,1,1,1). In the original ...


3

In Reinforcement Learning (RL), a reward function is part of the problem definition and should: Be based primarily on the goals of the agent. Take into account any combination of starting state $s$, action taken $a$, resulting state $s'$ and/or a random amount (a constant amount is just a random amount with a fixed value having probability 1). You should ...


3

Dennis Soemers provides an important point that from a theoretical standpoint, this can be seen as a non-issue. However, what you bring up is an important practical issue of potential-based reward shaping (PBRS). The issue is actually worse than you describe---it's more general than $s = s'$. In particular, the issue presents itself differently based on the ...


3

I don't think the situation you're sketching should be a problem at all. If $P(s)$ is high (e.g. $P(s) = 1000$), this means (according to your shaping / "heuristic") that it's valuable to be in the state $s$, that you expect to be able to get high future returns from that state. If you then continuously take actions that keep you in the same state $s$, it ...


3

Generally speaking, is it better for rewards to be a scalar, or is using matrices okay? Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and reinforcement learning (RL) methods. Although it is possible to accumulate matrices in various ways, by e.g. simple matrix addition, and come up with an analog for ...


2

The classic working reward scheme for two player zero sum games (i.e. if I win, you lose and vice versa) is simply: +1 for a win 0 for a draw -1 for a loss These rewards should be associated with the last move made by the player before the game is resolved. I thought about giving a negativ reward for the move played before the winning move. That is ...


2

Yes, you are right. It is somehow an arbitrary choice, although you should consider the reasonable numerical ranges of your activation functions if you decide to go beyond the values +/- 1. You can also have a think about whether you want to add a small reward for the agent reaching states that are near the goal, if you have an environment where such states ...


2

Sutton and Barto state, "The reward signal is your way of communicating to the robot [agent] what you want it to achieve, not how you want it achieved." Since you stated that the goal is to reach the finish line first, then a reward of $1$ for winning, $0$ for losing, and $0$ at all other time steps seems to fit that narrative. If a draw is identical to a ...


2

In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge. It may help if you think through what the actual consequences for making the "wrong" move are in your environment. There are a few self-consistent options: The move simply cannot be made and count as playing the ...


2

In reinforcement learning (RL), an immediate reward value must be returned after each action, along with the next state. This value can be zero though, which will have no direct impact on optimality or setting goals. Unless you are modifying the reward scheme to try and make an environment easier to learn (called reward shaping), then you should be aiming ...


2

I agree with Tomasz that the approach you are describing falls within the field of MORL. For a solid introduction MORL I would recommend the survey by Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67-113. https://www.jair.org/index....


2

The answer to both your concerns is: Add the previous action choice to the state representation. It is all you need to do. It gives the agent the data it needs to learn the association of negative reward from not matching the previous action. By making this data part of the state, you re-establish the Markov property in the MDP model of the environment, ...


1

In this scenario, is it preferable to create two separate environments based on the priority (input parameter) of the alert? It is difficult to make a hard rule here. If the resulting environments can be cleanly sorted into a few different categories, and the ideal behaviour and/or the states visited are radically different in each scenario, then maybe a ...


1

The reward function belongs the the environment and it is the only way the agent can explore the world given a state. If we want agent to do something specific, we must provide rewards to it in such a way that it will achieve our goals. It is thus very important that the reward function accurately indicates the exact behaviour. Depending on your goal you can ...


1

The reward function is up to you when you set the goals for the agent. If the goal is to score as highly as possible, before ending the game, then use the score. You may want to scale the score down if you are using neural networks, to prevent needing to handle very large error values in early phases of learning. If the goal is to win the game, and you do ...


1

Measure what you want to achieve as directly as possible, and reward that. Later you can add more sophisticated incentives for the type of motion etc, but the key to a good reward signal is that it measures the quality of a solution at a high level, without specifying how to achieve that solution. If you want your simulated car to explore, you will want to ...


1

I believe that there is no clear answer to your question. It essentially boils down to whether you are a reductionist – whether you believe that quantitative measurements can truly give justice to the complexity of the real world, and that a framework such as expectation maximization can losslessly capture what we care about as humans in the performing of ...


Only top voted, non community-wiki answers of a minimum length are eligible