# Tag Info

### Can Q-learning be used for continuous (state or action) spaces?

Q-learning for continuous state spaces Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the ...
• 32.7k
Accepted

### Can neural networks have continuous inputs and outputs, or do they have to be discrete?

Neural networks normally work in continuous spaces. A typical neural network function could be written as $f(\mathbf{x}, \mathbf{\theta}): \mathbb{R}^N \rightarrow \mathbb{R}^M$. That is, a function ...
• 32.7k
Accepted

### What techniques are used to make MDP discrete state space manageable?

tl:dr Read chapter 9 of an Introduction of Reinforcement Learning There is definitely a problem (a curse if you will) when the dimensionality of a task (MDP) grows. For fun, lets extend your problem ...
• 3,837
Accepted

### Can a large discrete action space be represented using Gaussian distributions?

The answer is "it depends". Once you have arranged the actions into order, a key trait is whether the action value function has a simple enough shape that sampling from a Gaussian policy ...
• 32.7k

### It is possible to solve a problem with continuous action spaces and no states with reinforcement learning?

A stateless RL problem can be reduced to a Multiarmed Bandit (MAB) problem. In such a scenario, taking an action will not change the state of the agent. So, this is the setting of a conventional MAB ...
Accepted

### How are continuous actions sampled (or generated) from the policy network in PPO?

As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :) A common example for continuous spaces is the reparameterization trick, ...
• 151
Accepted

### What is meant by a multi-dimensional continuous action space?

Let me rephrase it a little - it's a multidimensional continuous space of actions. So, you assign each action some vector from $R^{n}$. For intuition -- imagine you have a robot arm with four joints. ...
Accepted

Sounds like you have several problems with the way your policy is parametrized. You don't have to use the multivariate normal distribution. It can work, and probably others have done it already (if ...
• 1,281

1 vote
Accepted

### Reinforcement learning algorithms for large problems that are not based on a neural network

There are many state-of-the-art reinforcement learning algorithms for large problems with multidimensional continuous state spaces and actions. All of them rely on some sort of function approximator. ...
• 2,970
1 vote

### KL divergence coefficient update doesn't make sense in RLlib's PPO implementation

The loss in PPO has a penalty $$-\text{kl_coeff_val}\cdot\mathrm{KL}(\pi^{\text{NN}}||\pi^{\text{Target}}).$$ Increasing this coefficient means a lower reward, so it incentivizes a smaller KL ...
1 vote
Accepted

### How to define a continuous action distribution with a specific range for Reinforcement Learning?

First of all, the support of a normal distribution is the entire real line (or, in general, $\mathbb{R}^n$ for an $n$-dimensional multivariate normal distribution) so your action can be any number in \$...
• 4,920
1 vote

### What is meant by a multi-dimensional continuous action space?

The question has already been answered by Kirill, but I thought I'll add a good example of a multi-dimensional continuous action space too, namely the one I just encountered in the COBRA paper itself. ...
• 1,161
1 vote

### Can Q-learning be used for continuous (state or action) spaces?

Q-Learning for continuous state space Reinforcement learning algorithms (e.g Q-Learning) can be applied to both discrete and continuous spaces. If you understand how it works in discrete mode, then ...
• 579
1 vote

### If the average rewards start high and then decrease, could that indicate that the PPO is stuck at a local maximum?

I had the same problem where the reward kept decreasing and started to search for answers in the forum. I let the model trained while I search. As the model trained, the reward started to increase. ...
• 11

Only top scored, non community-wiki answers of a minimum length are eligible