8 votes

Can Q-learning be used for continuous (state or action) spaces?

Q-learning for continuous state spaces Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the ...
user avatar
  • 23.1k
6 votes
Accepted

Can neural networks have continuous inputs and outputs, or do they have to be discrete?

Neural networks normally work in continuous spaces. A typical neural network function could be written as $f(\mathbf{x}, \mathbf{\theta}): \mathbb{R}^N \rightarrow \mathbb{R}^M$. That is, a function ...
user avatar
  • 23.1k
6 votes
Accepted

What techniques are used to make MDP discrete state space manageable?

tl:dr Read chapter 9 of an Introduction of Reinforcement Learning There is definitely a problem (a curse if you will) when the dimensionality of a task (MDP) grows. For fun, lets extend your problem ...
user avatar
4 votes
Accepted

Can a large discrete action space be represented using Gaussian distributions?

The answer is "it depends". Once you have arranged the actions into order, a key trait is whether the action value function has a simple enough shape that sampling from a Gaussian policy ...
user avatar
  • 23.1k
3 votes

It is possible to solve a problem with continuous action spaces and no states with reinforcement learning?

A stateless RL problem can be reduced to a Multiarmed Bandit (MAB) problem. In such a scenario, taking an action will not change the state of the agent. So, this is the setting of a conventional MAB ...
user avatar
2 votes
Accepted

Policy Gradient ( Advantage actor-critic) for multiple simultaneous continuous actions

Sounds like you have several problems with the way your policy is parametrized. You don't have to use the multivariate normal distribution. It can work, and probably others have done it already (if ...
user avatar
  • 746
2 votes

How to have zero value or a value between 200 and 400 in the output of a deep learning model?

generally the approach is to have a separate head. For example, imagine you have latent vector $z_k$, you would output two values: $h(z_k)$ and $f(z_k)$ where $0 \leq h \leq 1$ and $b_0 \leq f \leq ...
user avatar
  • 2,229
2 votes
Accepted

How are continuous actions sampled (or generated) from the policy network in PPO?

As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :) A common example for continuous spaces is the reparameterization trick, ...
user avatar
  • 131
2 votes
Accepted

What is meant by a multi-dimensional continuous action space?

Let me rephrase it a little - it's a multidimensional continuous space of actions. So, you assign each action some vector from $R^{n}$. For intuition -- imagine you have a robot arm with four joints. ...
user avatar
1 vote
Accepted

Reinforcement learning algorithms for large problems that are not based on a neural network

There are many state-of-the-art reinforcement learning algorithms for large problems with multidimensional continuous state spaces and actions. All of them rely on some sort of function approximator. ...
user avatar
1 vote
Accepted

How to define a continuous action distribution with a specific range for Reinforcement Learning?

First of all, the support of a normal distribution is the entire real line (or, in general, $\mathbb{R}^n$ for an $n$-dimensional multivariate normal distribution) so your action can be any number in $...
user avatar
1 vote

What is meant by a multi-dimensional continuous action space?

The question has already been answered by Kirill, but I thought I'll add a good example of a multi-dimensional continuous action space too, namely the one I just encountered in the COBRA paper itself. ...
user avatar
1 vote

If the average rewards start high and then decrease, could that indicate that the PPO is stuck at a local maximum?

I had the same problem where the reward kept decreasing and started to search for answers in the forum. I let the model trained while I search. As the model trained, the reward started to increase. ...
user avatar
  • 11

Only top scored, non community-wiki answers of a minimum length are eligible