18 votes

Are there other approaches to deal with variable action spaces?

Does anyone know any paper regarding this subject? I'm not familiar with any off the top of my head. I do know that the vast majority of Reinforcement Learning literature focuses on settings with a ...
Dennis Soemers's user avatar
  • 10.2k
8 votes

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

You can try the actions yourselves, but if you want another reference, check out the documentation for ALE at GitHub. In particular, 0 means no action, 1 means fire, which is why they don't have an ...
ComputerScientist's user avatar
7 votes
Accepted

How to implement a variable action space in Proximal Policy Optimization?

The most straightforward solution is to simply make every action "legal", but implementing a consistent, deterministic mapping from potentially illegal actions to different legal actions. Whenever the ...
Dennis Soemers's user avatar
  • 10.2k
6 votes
Accepted

How does the Alpha Zero's move encoding work?

Let's do the code, so all the details are down. Encoding dictionary: ...
Kostya's user avatar
  • 2,516
4 votes

Are there other approaches to deal with variable action spaces?

Another approach that came across was to, assuming the number of different action set $n$ is quite small, have functions $f_{\theta_1}$, $f_{\theta_2}$, ..., $f_{\theta_n}$ that returns the action ...
maaartinus's user avatar
4 votes

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

You can try to figure out what exactly does an action do using such script: ...
Icyblade's user avatar
  • 141
4 votes
Accepted

Why does Deep Q Network outputs multiple Q values?

I think this was just a "clever" design choice. You can actually design a neural network (NN), to represent your Q function, which receives as input the state and an action and outputs the ...
nbro's user avatar
  • 40.2k
4 votes

Take action only at the beginning of the episode, not during each step

Should I make each step an episode so the action will be always applied at each step ? (i.e., should I consider a step the transition between the initial position of the balls and the their final ...
Neil Slater's user avatar
  • 31.7k
3 votes

Implementing an RL agent on a variable action space

I will try to answer your questions as best I can, potentially building on some of your ideas. Please note that I'm making certain assumptions about how to design the action space because I don't have ...
Cesar Ruiz's user avatar
3 votes

How to deal with a huge action space, where, at every step, there is a variable number of legal actions?

One way I can think of is to redefine "actions" in a game to make them more fragmented, in such a way that a player has multiple actions per turn. In chess, for example, we can define an action as ...
Bridgeburners's user avatar
3 votes
Accepted

How can I incorporate domain knowledge to choose actions in the case of large action spaces in multi-armed bandits?

You will want to look into Contextual Multi-Armed Bandits. These are MAB problems that additionally involve feature vectors in some way. You'll sometimes see researchers considering problems where ...
Dennis Soemers's user avatar
  • 10.2k
3 votes

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent?

There seems to be no difference between 2 & 4 and 3 & 5. The inconsistency mentioned by Icyblade is due to the mechanics of the Pong environment. "Each action is repeatedly performed for a ...
jaidmin's user avatar
  • 31
2 votes

How to implement a variable action space in Proximal Policy Optimization?

Change the action space at each step, depending on the internal_state. I assume this is nonsense. Yes, this seems overkill and makes the problem unnecessarily complex, there could be other things ...
Jithendaraa Subramanian's user avatar
2 votes

Are there RL techniques to deal with incremental action spaces?

Is there some solutions to update the model with only minor changes? In general, assuming the new action choices are meaningful - in at least some states, the expected return from taking one of the ...
Neil Slater's user avatar
  • 31.7k
2 votes
Accepted

How to use DQN when the action space can be different at different time steps?

There are two relevant neural network designs for DQN: Model q function directly $Q(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$, so neural network has concatenated input of state and ...
Neil Slater's user avatar
  • 31.7k
2 votes
Accepted

How should I define the action space for a card game like Magic: The Gathering?

There are several different ways you can model the state and action spaces in such sequential (extensive-form) environments/games. For environments with small action spaces or those typically ...
rhdxor's user avatar
  • 206
2 votes
Accepted

What is meant by a multi-dimensional continuous action space?

Let me rephrase it a little - it's a multidimensional continuous space of actions. So, you assign each action some vector from $R^{n}$. For intuition -- imagine you have a robot arm with four joints. ...
Kirill Fedyanin's user avatar
2 votes

How to handle large dimensionality differences between state and action inputs in a reinforcement learning predictor?

No the dimensionality size difference it's not a problem when defining a reward function using neural networks, but if you really want to do so, you can easily do that using two different mappings as ...
Alberto's user avatar
  • 1,677
1 vote

How do you define an action space for a card game with an unlimited and variable hand size?

A policy that handles an arbitrary number of actions can be specified as a function (think: neural network) that takes one specific action as input and outputs some kind of score. That score may be ...
maxy's user avatar
  • 223
1 vote
Accepted

What should the input and output of the Q-network be in the case of an ordinal action space?

Yes it is possible to use the action as input to neural network in DQN. For discrete actions represented as one-hot encoded features, the difference is minor: If all actions are in the output, your ...
Neil Slater's user avatar
  • 31.7k
1 vote

Is the agent aware of a possible different set of actions for each state?

This is actually an implementation choice, and will depend on how you chose to represent the agent's model of the function that maps from states to actions. If you explicitly represent the entire ...
John Doucette's user avatar
1 vote

How do you apply Q-learning when there are too many possible actions?

one of the downsides associated with the $Q$-Learning algorithm is that it must initialize a value $Q(s,a)$ for every $s\in S$ and every $a\in A$. If either one of your action space $A$ or state space ...
Hadar Sharvit's user avatar
1 vote

How to manage impossible actions?

You could code your agent's policy to never select impossible actions. Your other question implies that you are writing your own behaviour policy function (e.g. you asked about implementing a softmax ...
Neil Slater's user avatar
  • 31.7k
1 vote
Accepted

What's the benefit of repeating an action for a consecutive number of time steps?

A primary reason to repeat an action of a series of timesteps is that your environment may require more than one timestep to process the timestep. Said another way, changing the action every timestep ...
David Hoelzer's user avatar
1 vote
Accepted

How to define actions on a list of values?

I had tried working on a problem similar to this using combinatorial scoring games. I ran into other issues with the players competing, but I think I can give some advice to how I handled this. In my ...
Elfurd's user avatar
  • 46
1 vote
Accepted

Is it generally advisable to have a low dimensional action space in Reinforcement Learning?

Since the question may not be answered unambiguously in general, I will use the given example as a guide. As you correctly write, a large dimensionality leads to a very large solution space because of ...
dexteritas's user avatar
1 vote
Accepted

How to handle invalid actions for next state in Q-learning loss

do we also want to consider the subset of invalid actions for the $\max\limits_{a}Q(s_{t+1},a)$ No. Doing so would go against the theory behind the Bellman equation from which the update derives. The ...
Neil Slater's user avatar
  • 31.7k
1 vote
Accepted

How to incorporate action information in the state input of a DQN?

Drivers are not actions in this case, they are objects that are part of the state space, your state vector would look something like this \begin{equation} \mathbf{x} = [x_{o}^T, x_1^T,\ldots, x_N^T]^T ...
Brale's user avatar
  • 2,366
1 vote
Accepted

Why does each component of the tuple that represents an action have a categorical distribution in the TRPO paper?

I'm not sure specifically which Atari games present this type of action space, but you can imagine a game in which you can perform multiple different types of actions at the same timestep (i.e. the ...
mdc's user avatar
  • 380
1 vote

What is meant by a multi-dimensional continuous action space?

The question has already been answered by Kirill, but I thought I'll add a good example of a multi-dimensional continuous action space too, namely the one I just encountered in the COBRA paper itself. ...
stoic-santiago's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible