10
votes
Can Q-learning be used for continuous (state or action) spaces?
Q-learning for continuous state spaces
Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the ...
- 26.5k
6
votes
Accepted
Can neural networks have continuous inputs and outputs, or do they have to be discrete?
Neural networks normally work in continuous spaces. A typical neural network function could be written as $f(\mathbf{x}, \mathbf{\theta}): \mathbb{R}^N \rightarrow \mathbb{R}^M$. That is, a function ...
- 26.5k
6
votes
Accepted
What techniques are used to make MDP discrete state space manageable?
tl:dr
Read chapter 9 of an Introduction of Reinforcement Learning
There is definitely a problem (a curse if you will) when the dimensionality of a task (MDP) grows. For fun, lets extend your problem ...
- 3,697
4
votes
Accepted
Can a large discrete action space be represented using Gaussian distributions?
The answer is "it depends". Once you have arranged the actions into order, a key trait is whether the action value function has a simple enough shape that sampling from a Gaussian policy ...
- 26.5k
3
votes
It is possible to solve a problem with continuous action spaces and no states with reinforcement learning?
A stateless RL problem can be reduced to a Multiarmed Bandit (MAB) problem. In such a scenario, taking an action will not change the state of the agent.
So, this is the setting of a conventional MAB ...
2
votes
Accepted
What is meant by a multi-dimensional continuous action space?
Let me rephrase it a little - it's a multidimensional continuous space of actions. So, you assign each action some vector from $R^{n}$. For intuition -- imagine you have a robot arm with four joints. ...
- 549
2
votes
Accepted
How are continuous actions sampled (or generated) from the policy network in PPO?
As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :)
A common example for continuous spaces is the reparameterization trick, ...
- 141
2
votes
Accepted
Policy Gradient ( Advantage actor-critic) for multiple simultaneous continuous actions
Sounds like you have several problems with the way your policy is parametrized.
You don't have to use the multivariate normal distribution. It can work, and probably others have done it already (if ...
- 1,131
2
votes
How to have zero value or a value between 200 and 400 in the output of a deep learning model?
generally the approach is to have a separate head. For example, imagine you have latent vector $z_k$, you would output two values: $h(z_k)$ and $f(z_k)$ where $0 \leq h \leq 1$ and $b_0 \leq f \leq ...
- 2,339
1
vote
Accepted
What would be the Bellman optimality equation for $q_∗(s, a)$ for an MDP with continuous states and actions?
I think your equations are alright.
Anyway, this is just a question of mathematical notation.
In measure theory, a discrete random variable $X$ is said to have a counting measure over it's support $\...
- 36
1
vote
Accepted
Reinforcement learning algorithms for large problems that are not based on a neural network
There are many state-of-the-art reinforcement learning algorithms for large problems with multidimensional continuous state spaces and actions. All of them rely on some sort of function approximator.
...
- 2,005
1
vote
Accepted
How to define a continuous action distribution with a specific range for Reinforcement Learning?
First of all, the support of a normal distribution is the entire real line (or, in general, $\mathbb{R}^n$ for an $n$-dimensional multivariate normal distribution) so your action can be any number in $...
- 4,400
1
vote
What is meant by a multi-dimensional continuous action space?
The question has already been answered by Kirill, but I thought I'll add a good example of a multi-dimensional continuous action space too, namely the one I just encountered in the COBRA paper itself.
...
- 1,071
1
vote
Can Q-learning be used for continuous (state or action) spaces?
Q-Learning for continuous state space
Reinforcement learning algorithms (e.g Q-Learning) can be applied to both discrete and continuous spaces. If you understand how it works in discrete mode, then ...
- 499
1
vote
If the average rewards start high and then decrease, could that indicate that the PPO is stuck at a local maximum?
I had the same problem where the reward kept decreasing and started to search for answers in the forum.
I let the model trained while I search. As the model trained, the reward started to increase. ...
- 11
Only top scored, non community-wiki answers of a minimum length are eligible