Questions tagged [action-spaces]

For questions about action spaces in the context of reinforcement learning and other AI sub-fields.

Filter by
Sorted by
Tagged with
1 vote
1 answer
42 views

How to handle large dimensionality differences between state and action inputs in a reinforcement learning predictor?

I'm currently writing code for a reward predictor function r=f(s,a) in reinforcement learning, where 's' is the state with 256 dimensions (the embedding dimension after visual input is processed by an ...
XiaoBanni's user avatar
2 votes
1 answer
180 views

Implementing an RL agent on a variable action space

I've been trying to design and implement a DQN agent as an AI opponent option for a math game I've been working on. The game is a turn-based grid game, where players alternate making moves and ...
X496's user avatar
  • 21
0 votes
1 answer
125 views

How do you define an action space for a card game with an unlimited and variable hand size?

I'm new to the world of AI and have been primarily reading through the documentation for OpenAI's Gym/Gymnasium in hopes of training an AI to play a board game. One piece of information I haven't been ...
haec0007's user avatar
3 votes
1 answer
298 views

Take action only at the beginning of the episode, not during each step

I am working in an reinforcement learning environment with 1-dimensional action space. My action is only used at the first timestep of an episode and never again. In other word the action only affects ...
Optical_flow_lover's user avatar
1 vote
2 answers
198 views

How do you apply Q-learning when there are too many possible actions?

When the number of states in the Q-learning is large, we can refer to approximate Q-learning, but what should we do when we have a large number of actions?
znb's user avatar
  • 11
0 votes
0 answers
168 views

Can action space be part of the state space?

I'm working on a project where I have access to position coordinates and velocity components of multiple agents in an environment. Assuming that one agent is controllable while others are not ...
Eman.suradi's user avatar
2 votes
0 answers
130 views

Deep reinforcement learning the board game "Battle Sheep" - too large action space?

I was recently introduced to this simple board game called "Battle Sheep". In this game, two to four players try to acquire as many hex tiles from a hex grid as possible. You can find the ...
Jussi Hietanen's user avatar
0 votes
1 answer
194 views

How to manage impossible actions? [closed]

I am using Q-learning in julia language. Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I ...
Aquila's user avatar
  • 33
1 vote
0 answers
25 views

If we have a working reward function, would adding another action have a significant effect on the agent performance if task remains the same?

If we have a working reward function, providing the desired behavior and optimal policy in a continuous action/state-space problem, would adding another action significantly affect the possible ...
Philori's user avatar
  • 11
2 votes
0 answers
114 views

How to reduce the dimensionality of the actions in RL

I have a single-agent RL model in which the dimension of the dimension of the action space is $70$. This action space is too big and the deep RL agent is not learning properly. The boundaries of the ...
Leibniz's user avatar
  • 69
1 vote
0 answers
140 views

Is there a way use DQN to find the optimal combination of actions (control inputs) and environment parameters?

I am using DQN to find the optimal sequence of control inputs to a dynamic system. The setup is as follows: At the beginning of each episode, the system is initialized to the SAME initial condition ...
David Braun's user avatar
0 votes
1 answer
506 views

What's the benefit of repeating an action for a consecutive number of time steps?

What's the benefit of repeating an action for a consecutive number of time steps? Is there a way to tell if an agent in a given environment might perform better from repeated actions? I came across an ...
mugoh's user avatar
  • 531
0 votes
1 answer
218 views

How to define actions on a list of values?

For a DQN algorithm, where my state is a list of values, say: [5, 3, 4, 7, 8, 2, 6] How can I define an action space that allows me to move a value in the list from one position to another? For ...
sbk's user avatar
  • 3
3 votes
0 answers
223 views

How to deal with variable action ranges in RL for continuous action spaces

I am reading this paper on battery management using RL. The action consist in the charging/discharging power of the battery at timestep $t$. For instance, in the case of the charging power, the ...
Leibniz's user avatar
  • 69
1 vote
1 answer
457 views

Is it generally advisable to have a low dimensional action space in Reinforcement Learning?

In supervised or unsupervised learning, it is advised to reduce the dimensionality due to the curse of dimensionality in general. Is this also generally advisable for the action space of reinforcement ...
PeterBe's user avatar
  • 246
2 votes
1 answer
505 views

How to handle invalid actions for next state in Q-learning loss

I am implementing an RL application in an environment with illegal moves. For handling the illegal moves, I am currently just picking an action as the maximum Q-value from the set of legal Q-values. ...
John Rothman's user avatar
1 vote
1 answer
229 views

How to incorporate action information in the state input of a DQN?

I am working on an RL problem that I am trying to solve using a Deep Q-network. The problem concerns choosing drivers to take specific taxi orders. I am familiar with most of the existing works and ...
Stef's user avatar
  • 13
6 votes
1 answer
1k views

How does the Alpha Zero's move encoding work?

I am a beginner in AI. I'm trying to train a multi-agent RL algorithm to play chess. One issue that I ran into was representing the action space (legal moves/or honestly just moves in general) ...
Akshay Ghosh's user avatar
1 vote
0 answers
28 views

Is there any research on the application of policy gradients to problems where the selection of an action requires the selection of another one?

I am working on a problem and want to explore if it can be solved with PPO (or other policy gradient methods). The problem is that the action space is a bit special, compared to classic RL ...
Nathanson's user avatar
0 votes
0 answers
169 views

How should I model the state and action spaces for a problem where the goal is to draw a line between two points?

I have a problem where the goal is for the agent to draw a single line between two points on a $500 \times 500$ white image. I have built my DQN. For now, the output layer's size of the network is $[...
junfanbl's user avatar
  • 323
2 votes
1 answer
1k views

What should the input and output of the Q-network be in the case of an ordinal action space?

I recently started looking into implementations of the DQN algorithm (e.g. TensorFlow) in some more detail. All the implementations that I found use a network that gives an output for each possible ...
PontrYagin's user avatar
3 votes
1 answer
816 views

How to use DQN when the action space can be different at different time steps?

I would like to employ DQN to solve a constrained MDP problem. The problem has constraints on action space. At different time steps till the end, the available actions are different. It has different ...
ycenycute's user avatar
  • 341
4 votes
1 answer
1k views

How should I define the action space for a card game like Magic: The Gathering?

I'm trying to learn about reinforcement learning techniques. I have little background in machine learning from university, but never more than using a CNN on the MNIST database. My first project was ...
Why Not's user avatar
  • 43
0 votes
0 answers
147 views

How to implement RL model with increasing dimensions of state space and action space?

I've read in this discussion that "reinforcement learning is a way of finding the value function of a Markov Decision Process". I want to implement an RL model, whose state space and action ...
brzepkowski's user avatar
1 vote
1 answer
74 views

Why does each component of the tuple that represents an action have a categorical distribution in the TRPO paper?

I was going through the TRPO paper, and there was a line under Appendix D "Approximating Factored Policies with Neural Networks" in the last paragraph which I am unable to understand The ...
srij's user avatar
  • 13
2 votes
0 answers
82 views

When to do discretization to decrease the state/action space in RL?

When to do discretization to decrease the state/action space in RL? Can you give me some references that such a technique is used?
ddaedalus's user avatar
  • 919
1 vote
2 answers
956 views

What is meant by a multi-dimensional continuous action space?

In the context of Reinforcement Learning, what does it mean to have a multi-dimensional continuous action space? I came across the following in the COBRA Paper A method for learning a distribution ...
stoic-santiago's user avatar
1 vote
1 answer
234 views

How should I design the action space of an agent that needs to choose a 2d point and then shoot a cannonball?

I'm building a game environment (see the picture below) where an agent should position the mouse on the screen (see the coordinates on the upper right corner) and then click to shoot a cannonball. If ...
Voß's user avatar
  • 99
5 votes
1 answer
122 views

Is the agent aware of a possible different set of actions for each state?

I have a use case where the set of actions is different for different states. Is the agent aware of what actions are valid for each state, or is the agent only aware of the entire action space (in ...
Francis Chang's user avatar
6 votes
1 answer
349 views

Are there RL techniques to deal with incremental action spaces?

Let's say we have a problem that can be solved by some RL algorithms (DQN, for example, because we have discrete action space). At first, the action space is fixed (the number of actions is $n_1$), ...
user29643's user avatar
1 vote
1 answer
354 views

Should I ignore the actions RIGHTFIRE and LEFTFIRE in the SpaceInvaders environment? [closed]

I'm trying to replicate the DeepMind DQN paper. I'm using OpenAI's Gym. I'm trying to get a decent score with Space Invaders (using SpaceInvaders-v4 environment). I ...
JCP's user avatar
  • 173
18 votes
1 answer
6k views

How to deal with a huge action space, where, at every step, there is a variable number of legal actions?

I am working on creating an RL-based AI for a certain board game. Just as a general overview of the game so that you understand what it's all about: It's a discrete turn-based game with a board of ...
ytolochko's user avatar
  • 365
3 votes
1 answer
1k views

Why does Deep Q Network outputs multiple Q values?

I am learning Deep RL following this tutorial: https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8 I understand everything but one detail: This image shows ...
NMO's user avatar
  • 133
2 votes
2 answers
631 views

How can I design a reinforcement learning model for a game with multiple complex actions taken at a time?

I have a steady hex-map and turn-based wargame featuring WWII carrier battles. On a given turn, a player may choose to perform a large number of actions. Actions can be of many different types, and ...
Carrier Battles's user avatar
25 votes
2 answers
12k views

Are there other approaches to deal with variable action spaces?

This question is about Reinforcement Learning and variable action spaces for every/some states. Variable action space Let's say you have an MDP, where the number of actions varies between states (for ...
Rikard Olsson's user avatar
16 votes
3 answers
7k views

How to implement a variable action space in Proximal Policy Optimization?

I'm coding a Proximal Policy Optimization (PPO) agent with the Tensorforce library (which is built on top of TensorFlow). The first environment was very simple. Now, I'm diving into a more complex ...
Max's user avatar
  • 163
8 votes
0 answers
1k views

Is there a difference in the architecture of deep reinforcement learning when multiple actions are performed instead of a single action?

I've built a deep deterministic policy gradient reinforcement learning agent to be able to handle any games/tasks that have only one action. However, the agent seems to fail horribly when there are ...
Rui Nian's user avatar
  • 433
1 vote
1 answer
349 views

How can I incorporate domain knowledge to choose actions in the case of large action spaces in multi-armed bandits?

Suppose one is using a multi-armed bandit, and one has relatively few "pulls" (i.e. timesteps) relative to the action set. For example, maybe there are 200 timesteps and 100 possible actions....
wwl's user avatar
  • 153
6 votes
1 answer
3k views

How to deal with different actions for different states of the environment?

I'm new to this AI/Machine Learning and was playing around with OpenAI Gym a bit. When looking through the environments, I came across the Blackjack-v0 environment, ...
SomeDudeCalledMo's user avatar
10 votes
3 answers
17k views

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent? [closed]

Printing action_space for Pong-v0 gives Discrete(6) as output, i.e. $0, 1, 2, 3, 4, 5$ are actions defined in the environment as ...
cur10us's user avatar
  • 211