Questions tagged [action-spaces]
For questions about action spaces in the context of reinforcement learning and other AI sub-fields.
37
questions
0
votes
1
answer
96
views
How do you apply Q-learning when there are too many possible actions?
When the number of states in the Q-learning is large, we can refer to approximate Q-learning, but what should we do when we have a large number of actions?
- 1
0
votes
0
answers
52
views
Can action space be part of the state space?
I'm working on a project where I have access to position coordinates and velocity components of multiple agents in an environment. Assuming that one agent is controllable while others are not ...
- 133
2
votes
0
answers
90
views
Deep reinforcement learning the board game "Battle Sheep" - too large action space?
I was recently introduced to this simple board game called "Battle Sheep". In this game, two to four players try to acquire as many hex tiles from a hex grid as possible.
You can find the ...
0
votes
1
answer
116
views
How to manage impossible actions? [closed]
I am using Q-learning in julia language.
Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I ...
- 33
1
vote
0
answers
22
views
If we have a working reward function, would adding another action have a significant effect on the agent performance if task remains the same?
If we have a working reward function, providing the desired behavior and optimal policy in a continuous action/state-space problem, would adding another action significantly affect the possible ...
- 11
2
votes
0
answers
91
views
How to reduce the dimensionality of the actions in RL
I have a single-agent RL model in which the dimension of the dimension of the action space is $70$. This action space is too big and the deep RL agent is not learning properly. The boundaries of the ...
- 59
0
votes
0
answers
80
views
Can DDPG algorithm obtain time-dependent and time-independent actions simultaneously?
I am new to Reinforcement Learning. I have been working on a problem using Deep Deterministic Policy Gradient (DDPG).
I would like to know if it is possible to apply this algorithm to an optimization ...
1
vote
0
answers
96
views
Is there a way use DQN to find the optimal combination of actions (control inputs) and environment parameters?
I am using DQN to find the optimal sequence of control inputs to a dynamic system. The setup is as follows:
At the beginning of each episode, the system is initialized to the SAME initial condition ...
- 11
0
votes
1
answer
261
views
What's the benefit of repeating an action for a consecutive number of time steps?
What's the benefit of repeating an action for a consecutive number of time steps? Is there a way to tell if an agent in a given environment might perform better from repeated actions?
I came across an ...
- 511
0
votes
1
answer
117
views
How to define actions on a list of values?
For a DQN algorithm, where my state is a list of values, say:
[5, 3, 4, 7, 8, 2, 6]
How can I define an action space that allows me to move a value in the list from one position to another? For ...
- 3
2
votes
0
answers
141
views
How to deal with variable action ranges in RL for continuous action spaces
I am reading this paper on battery management using RL. The action consist in the charging/discharging power of the battery at timestep $t$. For instance, in the case of the charging power, the ...
- 59
1
vote
1
answer
272
views
Is it generally advisable to have a low dimensional action space in Reinforcement Learning?
In supervised or unsupervised learning, it is advised to reduce the dimensionality due to the curse of dimensionality in general.
Is this also generally advisable for the action space of reinforcement ...
- 212
2
votes
1
answer
327
views
How to handle invalid actions for next state in Q-learning loss
I am implementing an RL application in an environment with illegal moves. For handling the illegal moves, I am currently just picking an action as the maximum Q-value from the set of legal Q-values.
...
- 108
0
votes
1
answer
132
views
How to incorporate action information in the state input of a DQN?
I am working on an RL problem that I am trying to solve using a Deep Q-network. The problem concerns choosing drivers to take specific taxi orders. I am familiar with most of the existing works and ...
- 3
5
votes
1
answer
907
views
How does the Alpha Zero's move encoding work?
I am a beginner in AI. I'm trying to train a multi-agent RL algorithm to play chess. One issue that I ran into was representing the action space (legal moves/or honestly just moves in general) ...
- 95
1
vote
0
answers
26
views
Is there any research on the application of policy gradients to problems where the selection of an action requires the selection of another one?
I am working on a problem and want to explore if it can be solved with PPO (or other policy gradient methods). The problem is that the action space is a bit special, compared to classic RL ...
- 21
0
votes
0
answers
152
views
How should I model the state and action spaces for a problem where the goal is to draw a line between two points?
I have a problem where the goal is for the agent to draw a single line between two points on a $500 \times 500$ white image.
I have built my DQN. For now, the output layer's size of the network is $[...
- 323
2
votes
1
answer
938
views
What should the input and output of the Q-network be in the case of an ordinal action space?
I recently started looking into implementations of the DQN algorithm (e.g. TensorFlow) in some more detail. All the implementations that I found use a network that gives an output for each possible ...
- 23
3
votes
1
answer
574
views
How to use DQN when the action space can be different at different time steps?
I would like to employ DQN to solve a constrained MDP problem. The problem has constraints on action space. At different time steps till the end, the available actions are different. It has different ...
- 311
4
votes
1
answer
674
views
How should I define the action space for a card game like Magic: The Gathering?
I'm trying to learn about reinforcement learning techniques. I have little background in machine learning from university, but never more than using a CNN on the MNIST database.
My first project was ...
- 43
0
votes
0
answers
118
views
How to implement RL model with increasing dimensions of state space and action space?
I've read in this discussion that "reinforcement learning is a way of finding the value function of a Markov Decision Process".
I want to implement an RL model, whose state space and action ...
- 141
1
vote
1
answer
73
views
Why does each component of the tuple that represents an action have a categorical distribution in the TRPO paper?
I was going through the TRPO paper, and there was a line under Appendix D "Approximating Factored Policies with Neural Networks" in the last paragraph which I am unable to understand
The ...
- 13
2
votes
0
answers
73
views
When to do discretization to decrease the state/action space in RL?
When to do discretization to decrease the state/action space in RL? Can you give me some references that such a technique is used?
- 909
1
vote
2
answers
627
views
What is meant by a multi-dimensional continuous action space?
In the context of Reinforcement Learning, what does it mean to have a multi-dimensional continuous action space?
I came across the following in the COBRA Paper
A method for learning a distribution ...
- 1,051
1
vote
1
answer
188
views
How should I design the action space of an agent that needs to choose a 2d point and then shoot a cannonball?
I'm building a game environment (see the picture below) where an agent should position the mouse on the screen (see the coordinates on the upper right corner) and then click to shoot a cannonball. If ...
- 99
5
votes
1
answer
107
views
Is the agent aware of a possible different set of actions for each state?
I have a use case where the set of actions is different for different states. Is the agent aware of what actions are valid for each state, or is the agent only aware of the entire action space (in ...
6
votes
1
answer
297
views
Are there RL techniques to deal with incremental action spaces?
Let's say we have a problem that can be solved by some RL algorithms (DQN, for example, because we have discrete action space). At first, the action space is fixed (the number of actions is $n_1$), ...
- 61
1
vote
1
answer
282
views
Should I ignore the actions RIGHTFIRE and LEFTFIRE in the SpaceInvaders environment? [closed]
I'm trying to replicate the DeepMind DQN paper. I'm using OpenAI's Gym. I'm trying to get a decent score with Space Invaders (using SpaceInvaders-v4 environment). I ...
- 173
17
votes
1
answer
5k
views
How to deal with a huge action space, where, at every step, there is a variable number of legal actions?
I am working on creating an RL-based AI for a certain board game. Just as a general overview of the game so that you understand what it's all about: It's a discrete turn-based game with a board of ...
- 345
3
votes
1
answer
1k
views
Why does Deep Q Network outputs multiple Q values?
I am learning Deep RL following this tutorial: https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8
I understand everything but one detail:
This image shows ...
- 133
2
votes
2
answers
558
views
How can I design a reinforcement learning model for a game with multiple complex actions taken at a time?
I have a steady hex-map and turn-based wargame featuring WWII carrier battles.
On a given turn, a player may choose to perform a large number of actions. Actions can be of many different types, and ...
24
votes
2
answers
11k
views
Are there other approaches to deal with variable action spaces?
This question is about Reinforcement Learning and variable action spaces for every/some states.
Variable action space
Let's say you have an MDP, where the number of actions varies between states (for ...
- 341
16
votes
3
answers
7k
views
How to implement a variable action space in Proximal Policy Optimization?
I'm coding a Proximal Policy Optimization (PPO) agent with the Tensorforce library (which is built on top of TensorFlow).
The first environment was very simple. Now, I'm diving into a more complex ...
- 163
7
votes
0
answers
1k
views
Is there a difference in the architecture of deep reinforcement learning when multiple actions are performed instead of a single action?
I've built a deep deterministic policy gradient reinforcement learning agent to be able to handle any games/tasks that have only one action. However, the agent seems to fail horribly when there are ...
- 423
1
vote
1
answer
339
views
How can I incorporate domain knowledge to choose actions in the case of large action spaces in multi-armed bandits?
Suppose one is using a multi-armed bandit, and one has relatively few "pulls" (i.e. timesteps) relative to the action set. For example, maybe there are 200 timesteps and 100 possible actions....
- 153
4
votes
1
answer
2k
views
How to deal with different actions for different states of the environment?
I'm new to this AI/Machine Learning and was playing around with OpenAI Gym a bit. When looking through the environments, I came across the Blackjack-v0 environment, ...
10
votes
3
answers
17k
views
What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent? [closed]
Printing action_space for Pong-v0 gives Discrete(6) as output, i.e. $0, 1, 2, 3, 4, 5$ are actions defined in the environment as ...
- 211