Questions tagged [ddpg]

For questions related to the reinforcement learning algorithm called Deep Deterministic Policy Gradient (DDPG).

28 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8 votes
0 answers
1k views

Is there a difference in the architecture of deep reinforcement learning when multiple actions are performed instead of a single action?

I've built a deep deterministic policy gradient reinforcement learning agent to be able to handle any games/tasks that have only one action. However, the agent seems to fail horribly when there are ...
Rui Nian's user avatar
  • 433
4 votes
0 answers
396 views

What is the simplest policy gradient method to implement for a problem continuous action space?

I have a problem I would like to tackle with RL, but I am not sure if it is even doable. My agent has to figure out how to fill a very large vector (let's say from 600 to 4000 in the most complex ...
FS93's user avatar
  • 145
3 votes
0 answers
419 views

How to deal with a moving target in the Lunar Lander environment with DDPG?

I have noticed that DDPG does rather well at solving environments with a static target. For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
user1779362's user avatar
3 votes
0 answers
1k views

How does adding noise to the action in DDPG help in learning?

I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.
Ahmad Fares's user avatar
3 votes
0 answers
556 views

Should noise (such as OU) be decreased over time in actor / critic algorithms?

In most of RL algorithms I saw, there is a coefficient that reduces actions exploration over time, to help convergence. But in Actor-Critic, or other algorithms (A3C, DDPG, ...) used in continuous ...
Loheek's user avatar
  • 266
3 votes
0 answers
134 views

Can I use deterministic policy gradient methods for stochastic policy learning?

Can I treat a stochastic policy (over a finite action space of size $n$) as a deterministic policy (in the set of probability distribution in $\mathbb{R}^n$)? It seems to me that nothing is broken ...
ChubbyRuby's user avatar
2 votes
0 answers
1k views

Why is DDPG not learning and it does not converge?

I have used a different setting, but DDPG is not learning and it does not converge. I have used these codes 1,2, and 3 and I used different optimizers, activation functions, and learning rate but ...
I_Al-thamary's user avatar
2 votes
0 answers
352 views

A question about the Wolpertinger algorithm (Deep RL in Large Discrete Action Spaces paper)

I am trying to reproduce the recommender task experiment from this paper. The paper suggests to embed discrete actions into continuous action space and then to use the proposed Wolpertinger agent. ...
Pavel Korobov's user avatar
2 votes
0 answers
237 views

How can DDPG handle the discrete action space?

I am wondering how can DDPG or DPG handle the discrete action space. There are some papers saying that use Gumbel softmax with DDPG can make the discrete action problem be solved. However, will the ...
Jarvis's user avatar
  • 41
2 votes
0 answers
158 views

How to learn using DDPG in python solely using a timeseries datasets

I have a lengthy timeseries datasets which contains several variables (from sensors etc) to be classified as actions or states. Providing they are successfully done, I want to learn a control policy ...
JianNius's user avatar
  • 131
1 vote
0 answers
45 views

Role of $f$ Target Network in DDPG

I am trying to create a variant of DDPG in MATLAB that has no action-value $\langle Q \rangle$ net, but that instead works with networks $\langle V \rangle, \langle f \rangle, \langle r \rangle$, and ...
Vera Leighton 's user avatar
1 vote
0 answers
459 views

Why Soft Actor-Critic (SAC) uses a double Q trick?

Twin Delayed DDPG (TD3) uses a double Q trick since the policy is deterministic like in DDPG, which is to mitigate the maximum overestimation bias in DDPG. However, in SAC, the policy is stochastic, ...
Magi Feeney's user avatar
1 vote
0 answers
28 views

Why can't I train like a dataset of samples instead of maintaining replay buffer?

On observing the DDPG algorithm, we notice that the updation of neural networks is happening during the episode. But, it seems there is no issue if we allow the completion of an episode and then treat ...
hanugm's user avatar
  • 3,850
1 vote
0 answers
128 views

Could we add clipping in the output layer of the actor in DDPG?

I have a doubt about how clipping affects the training of the RL agents. In particular, I have come across a code for training DDPG agents, the pseudo-code is the following: ...
Leibniz's user avatar
  • 69
1 vote
0 answers
62 views

Setting initial values in DDPG to favor better actions

I'm working on a problem using DDPG. Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
jazz's user avatar
  • 11
1 vote
0 answers
271 views

How to parallelize multi-agent DDPG (MADDPG)

I am experimenting with MADDPG algorithm implemented in this repo. Since there were only a few agents (2-3) in the implementation (also in the original paper) steps like parameter updates, action ...
Mika's user avatar
  • 341
1 vote
0 answers
223 views

Gradual decrease in performance of a DDPG agent

I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some ...
Hirek Kubica's user avatar
1 vote
0 answers
338 views

How to deal with KerasRL DDPG algorithm getting stuck in a local optima?

I am using KerasRL DDPG to try to learn a policy on my own custom environment, but the agent is stuck in a local optima although I am adding the OrnsteinUhlenbeck randomization process. I used the ...
BAKYAC's user avatar
  • 36
1 vote
0 answers
207 views

Why would DDPG with Hindsight Experience Replay not converge?

I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
Vedant Shah's user avatar
1 vote
0 answers
88 views

Benchmarking SAC on Pybullet

So far I have seen TD3 and DDPG benchmarks on Pybullet environments, but I am looking for SAC benchmarks on Pybullet too, anyone can help?
ASA's user avatar
  • 151
1 vote
0 answers
32 views

What if the rewards induced by an environment are related to the policy too?

Assume we have a policy $\pi_{\theta}$ in a classic reinforcement learning setting, and a reward function $R^{\pi}(s,a)$ that changes as long as $\pi$ changes i.e. not only is it predefined by the ...
ASA's user avatar
  • 151
1 vote
0 answers
56 views

Why does the result when restoring a saved DDPG model differ significantly from the result when saving it?

I save the trained model after a certain number of episodes with the special save() function of the DDPG class (the network is saved when the reward reaches zero), but when I restore the model again ...
Ne1zvestnyj's user avatar
1 vote
0 answers
106 views

DDPG: how to implement continuous action space bounded in the interval [-2, 2]?

I am a newbie in reinforcement learning and trying to understand how to implement continuous actions bounded by $[-2, 2]$. My research shows that doing nothing is a possible solution (i.e. action of 4....
TFbie's user avatar
  • 73
1 vote
0 answers
47 views

Should we multiply the target of actor by the importance sampling ratio when prioritized replay is applied to DDPG?

According to PER, we have to multiply the $Q$ error $\delta_i$ by the importance sampling ratio to correct the bias introduced by the imbalance sampling of PER, where importance sampling ratio is ...
Maybe's user avatar
  • 471
0 votes
0 answers
28 views

DDPG input incompatible in OpenAI Gym custom environment

I have a custom OpenAI Gym Boid flocking environment using StableBAselines3 for DDPG. This error I encounter occurred previously due to wrong input size of actions. But my reset function is correct, ...
Hamza's user avatar
  • 1
0 votes
0 answers
174 views

Unable to interpret DDPG actor-critic loss curves

I am training a DDPG actor-critic agent and ploting rewards and loss curves each episode to track the training evolution. Rewards values in the plot correspond to the total reward per episode divided ...
davipeix's user avatar
0 votes
0 answers
54 views

MADDPG: reward drops fast after updating ac network

I'm using MADDPG to do computation offloading. But the reward drops fast once I start sampling from the replay buffer. ...
hhhhhh's user avatar
  • 1
0 votes
0 answers
800 views

Which is the best RL algo for continuous states but discrete action spaces problem

I am trying to train an AI with an environment where the states are continuous but the actions are discrete, that means I can not apply DDPG or TD3. Can someone please help to let know what should be ...
user2783767's user avatar