# Tag Info

14

Does anyone know any paper regarding this subject? I'm not familiar with any off the top of my head... I do know that the vast majority of Reinforcement Learning literature focuses on settings with a fixed action space (like robotics where your actions determine how you attempt to move / rotate a particular part of the robot, or simple games where you ...

7

You can try the actions yourselves, but if you want another reference, check out the documentation for ALE at GitHub. In particular, 0 means no action, 1 means fire, which is why they don't have an effect on the racket. Here's a better way: env.unwrapped.get_action_meanings()

6

The most straightforward solution is to simply make every action "legal", but implementing a consistent, deterministic mapping from potentially illegal actions to different legal actions. Whenever the PPO implementation you are using selects an illegal action, you simply replace it with the legal action that it maps to. Your PPO algorithm can then still ...

4

I think this was just a "clever" design choice. You can actually design a neural network (NN), to represent your Q function, which receives as input the state and an action and outputs the corresponding Q value. However, to obtain $\max_aQ(s', a)$ (which is a term of the update rule of the Q-learning algorithm) you would need a "forward pass" of this network ...

3

(3) Another approach that came across was to, assuming the number of different action set $n$ is quite small, have functions $f_{\theta_1}$, $f_{\theta_2}$, ..., $f_{\theta_n}$ that returns the action regarding that perticular state with $n$ valid actions. E.i, the performed action of a state $s$ with 3 number of actions will be predicted by $\underset{a}{\... 3 You can try to figure out what exactly does an action do using such script: action = 0 # modify this! o = env.reset() for i in xrange(5): # repeat one action for five times o = env.step(action)[0] IPython.display.display( Image.fromarray( o[:,140:142] # extract your bat ).resize((300, 300)) # bigger image, easy for visualization ) ... 3 You will want to look into Contextual Multi-Armed Bandits. These are MAB problems that additionally involve feature vectors in some way. You'll sometimes see researchers considering problems where you get to see a single feature vector per timestep (like an "environment state" you're in) which may provide useful information. You'll also sometimes see ... 2 There seems to be no difference between 2 & 4 and 3 & 5. The inconsistency mentioned by Icyblade is due to the mechanics of the Pong environment. "Each action is repeatedly performed for a duration of k frames, where k is uniformly sampled from {2,3,4}" So the action is just repeated a different number of times due to randomness 2 Change the action space at each step, depending on the internal_state. I assume this is nonsense. Yes, this seems overkill and makes the problem unnecessarily complex, there could be other things you can do. Do nothing : let the model understand that choosing an unavailable action has no impact. While this will not harm your model negatively, in any way ... 2 Is there some solutions to update the model with only minor changes? In general, assuming the new action choices are meaningful - in at least some states, the expected return from taking one of the new actions is higher than the current optimal policy using just the old action selection - then the answer here is "no". At the very least you will need to re-... 2 Let me rephrase it a little - it's a multidimensional continuous space of actions. So, you assign each action some vector from$R^{n}$. For intuition -- imagine you have a robot arm with four joints. For every joint you could applied a rotation force from [-1, 1] and thus you get a 4-D vector with float numbers for each possible action. 2 There are two relevant neural network designs for DQN: Model q function directly$Q(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$, so neural network has concatenated input of state and action, and outputs a single real value. This is arguably the more natural fit to Q learning, but can be inefficient. Model all q values for given state$Q(s,\...

2

Let's do the code, so all the details are down. Encoding dictionary: codes, i = {}, 0 for nSquares in range(1,8): for direction in ["N", "NE", "E", "SE", "S", "SW", "W", "NW"]: codes[(nSquares,direction)] = i i += 1 You'll see that the codes dictionary will ...

1

Yes it is possible to use the action as input to neural network in DQN. For discrete actions represented as one-hot encoded features, the difference is minor: If all actions are in the output, your neural network function is $f(s): \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|} = [\hat{q}(s,a_1), \hat{q}(s,a_2), \hat{q}(s,a_3) ...]$, and you take the ...

1

There are several different ways you can model the state and action spaces in such sequential (extensive-form) environments/games. For environments with small action spaces or those typically introduced to beginning-RL students, the state space and action space remains constant along an agent's trajectory (termed normal form games when there are multiple ...

1

The question has already been answered by Kirill, but I thought I'll add a good example of a multi-dimensional continuous action space too, namely the one I just encountered in the COBRA paper itself. In all of our experiments we use a 2-dimensional virtual "touch-screen" environment that contains objects with configurable shape, position, and ...

1

This is actually an implementation choice, and will depend on how you chose to represent the agent's model of the function that maps from states to actions. If you explicitly represent the entire state space, as you might chose to do with simple benchmark problems that you solve by directly solving an MDP with something like value iteration, then you can ...

1

One way I can think of is to redefine "actions" in a game to make them more fragmented, in such a way that a player has multiple actions per turn. In chess, for example, we can define an action as choosing a tile from which to move, or choosing the motion from the chosen tile, as 2 separate actions. As an example a turn might consist of the following two ...

1

The master's thesis Action space representation in combinatorial multi-armed bandits (2015) seems to provide an answer to my question. Several algorithms can be used Naive Monte-Carlo Sampling (NMC) Linear Side Information (LSI) Monte Carlo Tree Search with Hierarchical Expansion (MCTS-HE) MCTS with Dimensional Expansion The idea is to divide and conquer. ...

1

Normally, the set of actions that the agent can execute does not change over time, but some actions can become impossible in different states (for example, not every move is possible in any position of the TicTacToe game). Take a look as example at pice of code https://github.com/haje01/gym-tictactoe/blob/master/examples/base_agent.py : ava_actions = env....

Only top voted, non community-wiki answers of a minimum length are eligible