Questions tagged [ddpg]
For questions related to the reinforcement learning algorithm called Deep Deterministic Policy Gradient (DDPG).
28
questions with no upvoted or accepted answers
7
votes
0
answers
1k
views
Is there a difference in the architecture of deep reinforcement learning when multiple actions are performed instead of a single action?
I've built a deep deterministic policy gradient reinforcement learning agent to be able to handle any games/tasks that have only one action. However, the agent seems to fail horribly when there are ...
- 423
4
votes
0
answers
377
views
What is the simplest policy gradient method to implement for a problem continuous action space?
I have a problem I would like to tackle with RL, but I am not sure if it is even doable.
My agent has to figure out how to fill a very large vector (let's say from 600 to 4000 in the most complex ...
- 145
3
votes
0
answers
294
views
How to deal with a moving target in the Lunar Lander environment with DDPG?
I have noticed that DDPG does rather well at solving environments with a static target.
For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
- 131
3
votes
0
answers
744
views
How does adding noise to the action in DDPG help in learning?
I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.
- 31
3
votes
0
answers
464
views
Should noise (such as OU) be decreased over time in actor / critic algorithms?
In most of RL algorithms I saw, there is a coefficient that reduces actions exploration over time, to help convergence.
But in Actor-Critic, or other algorithms (A3C, DDPG, ...) used in continuous ...
- 266
3
votes
0
answers
131
views
Can I use deterministic policy gradient methods for stochastic policy learning?
Can I treat a stochastic policy (over a finite action space of size $n$) as a deterministic policy (in the set of probability distribution in $\mathbb{R}^n$)?
It seems to me that nothing is broken ...
- 31
2
votes
0
answers
846
views
Why is DDPG not learning and it does not converge?
I have used a different setting, but DDPG is not learning and it does not converge. I have used these codes 1,2, and 3 and I used different optimizers, activation functions, and learning rate but ...
- 52
2
votes
0
answers
196
views
How can DDPG handle the discrete action space?
I am wondering how can DDPG or DPG handle the discrete action space. There are some papers saying that use Gumbel softmax with DDPG can make the discrete action problem be solved. However, will the ...
- 41
2
votes
0
answers
144
views
How to learn using DDPG in python solely using a timeseries datasets
I have a lengthy timeseries datasets which contains several variables (from sensors etc) to be classified as actions or states. Providing they are successfully done, I want to learn a control policy ...
- 131
1
vote
0
answers
32
views
Role of $f$ Target Network in DDPG
I am trying to create a variant of DDPG in MATLAB that has no action-value $\langle Q \rangle$ net, but that instead works with networks $\langle V \rangle, \langle f \rangle, \langle r \rangle$, and ...
1
vote
0
answers
207
views
Why Soft Actor-Critic (SAC) uses a double Q trick?
Twin Delayed DDPG (TD3) uses a double Q trick since the policy is deterministic like in DDPG, which is to mitigate the maximum overestimation bias in DDPG. However, in SAC, the policy is stochastic, ...
- 41
1
vote
0
answers
27
views
Why can't I train like a dataset of samples instead of maintaining replay buffer?
On observing the DDPG algorithm, we notice that the updation of neural networks is happening during the episode.
But, it seems there is no issue if we allow the completion of an episode and then treat ...
- 3,531
1
vote
0
answers
102
views
Could we add clipping in the output layer of the actor in DDPG?
I have a doubt about how clipping affects the training of the RL agents.
In particular, I have come across a code for training DDPG agents, the pseudo-code is the following:
...
- 69
1
vote
0
answers
55
views
Setting initial values in DDPG to favor better actions
I'm working on a problem using DDPG.
Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
- 11
1
vote
0
answers
175
views
How to parallelize multi-agent DDPG (MADDPG)
I am experimenting with MADDPG algorithm implemented in this repo. Since there were only a few agents (2-3) in the implementation (also in the original paper) steps like parameter updates, action ...
- 331
1
vote
0
answers
166
views
Gradual decrease in performance of a DDPG agent
I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some ...
- 11
1
vote
0
answers
240
views
How to deal with KerasRL DDPG algorithm getting stuck in a local optima?
I am using KerasRL DDPG to try to learn a policy on my own custom environment, but the agent is stuck in a local optima although I am adding the OrnsteinUhlenbeck randomization process. I used the ...
- 36
1
vote
0
answers
165
views
Why would DDPG with Hindsight Experience Replay not converge?
I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
- 105
1
vote
0
answers
81
views
Benchmarking SAC on Pybullet
So far I have seen TD3 and DDPG benchmarks on Pybullet environments, but I am looking for SAC benchmarks on Pybullet too, anyone can help?
- 151
1
vote
0
answers
297
views
A question about the Wolpertinger algorithm (Deep RL in Large Discrete Action Spaces paper)
I am trying to reproduce the recommender task experiment from this paper.
The paper suggests to embed discrete actions into continuous action space and then to use the proposed Wolpertinger agent.
...
- 11
1
vote
0
answers
31
views
What if the rewards induced by an environment are related to the policy too?
Assume we have a policy $\pi_{\theta}$ in a classic reinforcement learning setting, and a reward function $R^{\pi}(s,a)$ that changes as long as $\pi$ changes i.e. not only is it predefined by the ...
- 151
1
vote
0
answers
52
views
Why does the result when restoring a saved DDPG model differ significantly from the result when saving it?
I save the trained model after a certain number of episodes with the special save() function of the DDPG class (the network is saved when the reward reaches zero), but when I restore the model again ...
- 11
1
vote
0
answers
91
views
DDPG: how to implement continuous action space bounded in the interval [-2, 2]?
I am a newbie in reinforcement learning and trying to understand how to implement continuous actions bounded by $[-2, 2]$. My research shows that doing nothing is a possible solution (i.e. action of 4....
- 73
1
vote
0
answers
36
views
Should we multiply the target of actor by the importance sampling ratio when prioritized replay is applied to DDPG?
According to PER, we have to multiply the $Q$ error $\delta_i$ by the importance sampling ratio to correct the bias introduced by the imbalance sampling of PER, where importance sampling ratio is ...
- 441
0
votes
0
answers
35
views
What happens if I don't do action exploration in DDPG?
Consider the following line from the pseudocode of the DDPG algorithm
Select action $a_t = \mu(s_t| \theta_\mu) + \mathcal{N}_t$ according to the current policy and exploration noise
If I replace ...
- 3,531
0
votes
0
answers
275
views
Why DDPG losses don't decrease while the reward grows?
I've noticed that training a DDPG agent in the Reacher-v2 environment of OpenAI Gym, the losses of both actor and critic first decrease but after a while start increasing but the episode mean reward ...
- 41
0
votes
0
answers
86
views
What method to use when optimizing an array of data
Say I have an array of data, where each element describes a shape made of points, in vector form (each vector has several hundred dimensions). Each element also has a rating that gets higher, the ...
0
votes
0
answers
614
views
Which is the best RL algo for continuous states but discrete action spaces problem
I am trying to train an AI with an environment where the states are continuous but the actions are discrete, that means I can not apply DDPG or TD3.
Can someone please help to let know what should be ...
- 121