Questions tagged [continuous-action-spaces]

For questions about continuous action spaces in the context of reinforcement learning (or other artificial intelligence sub-fields). There is also the tag for discrete action spaces.

Filter by
Sorted by
Tagged with
0 votes
0 answers
35 views

How to apply DRL to solve a problem that involves mixed discrete-continuous action spaces where the action's size changes over time?

I have a reinforcement learning problem where a possible action is a probability vector $[p_1\ldots,p_n]$ of size $n\in\{1,\ldots,N\}$, where each element $p_i$ of the vector is between $0$ and $1$ ...
zdm's user avatar
  • 301
3 votes
0 answers
33 views

Algorithms for average reward reinforcement learning in continuous/general state-action space

I see that discounted reward reinforcement learning has been extensively studied in the literature. However, the average reward metric receives less attention, and it looks like algorithms for this ...
k2pctdn's user avatar
  • 55
0 votes
0 answers
28 views

Help defining environment with complex action space

I'm working on a personal MARL project with a high-dimensional and continuous action space. The environment is designed to give positive rewards to actions between some moving limits of the action ...
Sebastian Tinoco's user avatar
1 vote
1 answer
79 views

Model-based learning in continuous state and action spaces

I am interested in learning how transition probabilities/mdps are constructed in continuous state and action space model-based learning setting. There is some literature available on this matter, but ...
hogger's user avatar
  • 11
1 vote
1 answer
93 views

RL - Can RL be applied to problems where the next state is not the next observation?

I'm quite new on the study of reinforcement learning, and Im working on a communication problem with continuous large actions range for my final graduation work. I'm trying to use Gaussian Policy and ...
MaarcosNascimen's user avatar
1 vote
1 answer
138 views

What would be the Bellman optimality equation for $q_∗(s, a)$ for an MDP with continuous states and actions?

I'm currently studying Reinforcement Learning and I'd like to know what would be the Bellman optimality equation for action values $q_∗(s, a)$ for a MDP with continuous states and actions, written out ...
user's user avatar
  • 145
1 vote
0 answers
85 views

How can I get an integer as output for continuous action space PPO reinforcement learning?

I have a huge discrete action space, the learning stability is not good. I'd like to move to continuous action space but the only output for my task can be a positive integer (let's say in the range 0 ...
NAnn's user avatar
  • 111
1 vote
0 answers
48 views

What Kind of Reinforcement Learning Algorithms Can Be Used when the Action Space is Unfeasibly Large?

I know Deep Q network as a $S\times A$ DNN which maps the $S$ dimensional statespace to q-values of $A$ distinct actions. In my problem, the action space is still discrete, and finite, but depending ...
Della's user avatar
  • 111
1 vote
1 answer
131 views

Reinforcement learning algorithms for large problems that are not based on a neural network

I have a large control problem with multidimensional continuous inputs (13) and outputs (3). I tried several Reinforcement learning algorithms like Deep-Q-Networks (DQN), Proximal Policy Optimization (...
PeterBe's user avatar
  • 246
3 votes
0 answers
223 views

How to deal with variable action ranges in RL for continuous action spaces

I am reading this paper on battery management using RL. The action consist in the charging/discharging power of the battery at timestep $t$. For instance, in the case of the charging power, the ...
Leibniz's user avatar
  • 69
0 votes
0 answers
104 views

Training a RL agent using different data at each episode

I am training a RL agent whose state is composed of two numbers, ranging between 4 ~ 16 and 0 ~ 360. The action is continuous and between 0~90. In real life, the states can be any I am training a TD3 ...
Leibniz's user avatar
  • 69
3 votes
1 answer
3k views

Can neural networks have continuous inputs and outputs, or do they have to be discrete?

In general, can ANNs have continuous inputs and outputs, or do they have to be discrete? So, basically, I would like to have a mapping of continuous inputs to continuous outputs. Is this possible? ...
PeterBe's user avatar
  • 246
0 votes
2 answers
831 views

KL divergence coefficient update doesn't make sense in RLlib's PPO implementation

I am using RLlib (Ray 1.4.0)'s implementation of PPO for a multi-agent scenario with continuous actions, and I find that the loss includes the KL divergence penalty term, apart from the surrogate loss,...
hridayns's user avatar
  • 233
1 vote
0 answers
27 views

Is there a gentle introduction to reinforcement learning applied to MDPs with continuous state spaces?

I am looking for a gentle introduction (videos, lecture notes, tutorials, books) on reinforcement learning (MDPs) involving continuous states (or very large cardinality of state space). In particular, ...
cgo's user avatar
  • 175
1 vote
1 answer
1k views

How to define a continuous action distribution with a specific range for Reinforcement Learning?

Specifically for continuous control PPO, let's say my action space range is between $X$ (low) and $Y$ (high) and they are all sampled from a Gaussian Action Distribution with mean $\mu$ and standard ...
hridayns's user avatar
  • 233
0 votes
0 answers
584 views

RLlib's Multi-agent PPO continuous actions turn into nan

After some amount of training on a custom Multi-agent sparse-reward environment using RLlib's (1.4.0) PPO network, I found that my continuous actions turn into nan (explodes?) which is probably caused ...
hridayns's user avatar
  • 233
-1 votes
1 answer
623 views

Policy Gradient ( Advantage actor-critic) for multiple simultaneous continuous actions

i'm trying to solve a problem in which i need to carry out reinforcement learning with multiple simultaneous actions in continuous action space . i checked the multiagent structure; however, im trying ...
navid mohamadi's user avatar
1 vote
0 answers
962 views

PPO in continuous control not working

I have PPO agent for discrete action space for LunarLander-v2 env in gym and it works well. However, when i am trying to solve continuous version of the same env - <...
Alexander Yukhimchuk's user avatar
1 vote
1 answer
26 views

How to have zero value or a value between 200 and 400 in the output of a deep learning model?

I want to implement a DDPG method and obviously, the action space will be continuous. I have three outputs. The first output should be zero or a value between 200 and 400, and the other outputs have ...
Jacksss's user avatar
  • 11
4 votes
1 answer
672 views

Can a large discrete action space be represented using Gaussian distributions?

I have a large 1D action space, e.g. dim(A)=2000-10000. Can I use continuous action space where I could learn the mean and std of the Gaussian distributions that I would use to sample action from and ...
Mika's user avatar
  • 341
1 vote
0 answers
96 views

Which RL algorithm would be suitable for this multi-dimensional and continuous action space?

Is there an RL approach/algorithm that would be suited for the following kind of problem? There is a continuous action space with an action value $A_{a,t}$ for each action dimension $a$. The ...
shimeji42's user avatar
  • 111
1 vote
0 answers
313 views

How to make SAC (Soft-Actor-Critic) learn a policy?

I cannot make SAC learn a task in a certain environment. The point is that it actually sometimes finds a very good policy, but it never learns the policy in the end. I am using the SAC implementation ...
vega's user avatar
  • 11
1 vote
0 answers
507 views

Are actions deterministic during testing in continuous action space PPO?

In a continuous action space (for instance, in PPO, TRPO, REINFORCE, etc.), during training, an action is sampled from the random distribution with $\mu$ and $\sigma$. This results in an inherent ...
Mika's user avatar
  • 341
6 votes
1 answer
3k views

How are continuous actions sampled (or generated) from the policy network in PPO?

I am trying to understand and reproduce the Proximal Policy Optimization (PPO) algorithm in detail. One thing that I find missing in the paper introducing the algorithm is how exactly actions $a_t$ ...
Daniel B.'s user avatar
  • 815
1 vote
0 answers
460 views

What would be the AlphaGo's performance in continuous action space?

During my research for Google DeepMind's Go-playing program Alpha Go and its successor Alpha Go Zero, I discovered that the system uses a clever pipeline and an interplay of blocks of both policy and ...
maven's user avatar
  • 31
1 vote
0 answers
405 views

What adapts an algorithm to continuous or to discrete action spaces?

Some RL algorithms can only be used for environments with continuous action spaces (e.g TD3, SAC), while others only for discrete action spaces (DQN), and some for both REINFORCE and other policy ...
mugoh's user avatar
  • 531
1 vote
2 answers
958 views

What is meant by a multi-dimensional continuous action space?

In the context of Reinforcement Learning, what does it mean to have a multi-dimensional continuous action space? I came across the following in the COBRA Paper A method for learning a distribution ...
stoic-santiago's user avatar
1 vote
0 answers
105 views

DDPG: how to implement continuous action space bounded in the interval [-2, 2]?

I am a newbie in reinforcement learning and trying to understand how to implement continuous actions bounded by $[-2, 2]$. My research shows that doing nothing is a possible solution (i.e. action of 4....
TFbie's user avatar
  • 73
4 votes
0 answers
393 views

What is the simplest policy gradient method to implement for a problem continuous action space?

I have a problem I would like to tackle with RL, but I am not sure if it is even doable. My agent has to figure out how to fill a very large vector (let's say from 600 to 4000 in the most complex ...
FS93's user avatar
  • 145
5 votes
1 answer
375 views

It is possible to solve a problem with continuous action spaces and no states with reinforcement learning?

I want to use Reinforcement Learning to optimize the distribution of energy for a peak shaving problem given by a thermodynamical simulation. However, I am not sure how to proceed as the action space ...
FS93's user avatar
  • 145
19 votes
2 answers
19k views

Can Q-learning be used for continuous (state or action) spaces?

Many examples work with a table-based method for Q-learning. This may be suitable for a discrete state (observation) or action space, like a robot in a grid world, but is there a way to use Q-learning ...
Bryan McGill's user avatar
1 vote
1 answer
2k views

If the average rewards start high and then decrease, could that indicate that the PPO is stuck at a local maximum?

I'm trying to train a PPO agent in a 3D balance ball environment. My action space is continuous. In the following graph, each dot shows the average reward from 100 episodes. Could this graph indicate ...
Tony Ho's user avatar
  • 11
1 vote
0 answers
114 views

How to generalize finite MDP to general MDP?

Suppose, for simplicity sake, to be in a discrete time domain with the action set being the same for all states $S \in \mathcal{S}$. Thus, in a finite Markov Decision Process, the sets $\mathcal{A}$, $...
gvgramazio's user avatar
6 votes
1 answer
629 views

What techniques are used to make MDP discrete state space manageable?

Generating a discretized state space for an MDP (Markov Decision Process) model seems to suffer from the curse of dimensionality. Supposed my state has a few simple features: Feeling: Happy/Neutral/...
Brendan Hill's user avatar