Questions tagged [ddpg]

For questions related to the reinforcement learning algorithm called Deep Deterministic Policy Gradient (DDPG).

Filter by
Sorted by
Tagged with
0 votes
0 answers
52 views

Unable to interpret DDPG actor-critic loss curves

I am training a DDPG actor-critic agent and ploting rewards and loss curves each episode to track the training evolution. Rewards values in the plot correspond to the total reward per episode divided ...
davipeix's user avatar
0 votes
0 answers
42 views

MADDPG: reward drops fast after updating ac network

I'm using MADDPG to do computation offloading. But the reward drops fast once I start sampling from the replay buffer. ...
hhhhhh's user avatar
  • 1
1 vote
0 answers
40 views

Role of $f$ Target Network in DDPG

I am trying to create a variant of DDPG in MATLAB that has no action-value $\langle Q \rangle$ net, but that instead works with networks $\langle V \rangle, \langle f \rangle, \langle r \rangle$, and ...
Vera Leighton 's user avatar
1 vote
0 answers
348 views

Why Soft Actor-Critic (SAC) uses a double Q trick?

Twin Delayed DDPG (TD3) uses a double Q trick since the policy is deterministic like in DDPG, which is to mitigate the maximum overestimation bias in DDPG. However, in SAC, the policy is stochastic, ...
Magi Feeney's user avatar
1 vote
2 answers
256 views

RL solutions for OpenAI Gym environments?

Is there any place where people share their agent's settings for solving OpenAI Gym Environments? For example, I'd like to know what are good parameters for a DDPG agent to learn the task in Reacher-...
pippo's user avatar
  • 41
1 vote
0 answers
27 views

Why can't I train like a dataset of samples instead of maintaining replay buffer?

On observing the DDPG algorithm, we notice that the updation of neural networks is happening during the episode. But, it seems there is no issue if we allow the completion of an episode and then treat ...
hanugm's user avatar
  • 3,612
2 votes
1 answer
96 views

How to sample the tuples during the initial time steps of the DDPG algorithm?

I am facing an issue in understanding the following line from the pseudocode of the DDPG algorithm Sample a random minibatch of $N$ transitions $(s_i, a_i, r_i, s_{i+1})$ from $R$ Here $N$ is a ...
hanugm's user avatar
  • 3,612
1 vote
0 answers
109 views

DDPG agent with convolutional layers for feature extraction [closed]

I'm trying to come up with a definition of the critic for a DDPG agent in PyTorch using a CNN as a feature extractor. It is pretty straight forward for the actor model. However, for the critic model I ...
Andreas Karatzas's user avatar
0 votes
1 answer
132 views

Replay buffer action range in DDPG

I have an environment where the agent action is in range [0, 1.57]. My actor network in DDPG has a tanh activation, and so the network values are in the range [-1,1]. Hence I change the scaling from [-...
goldfinch's user avatar
0 votes
1 answer
403 views

Why does my actor-critic network always give either -1 or 1 at the output layer?

I have an actor-critic network. The state space contains continuous variables with different ranges like (0,1.57) and (-0.70, 0.70). And it also contain absolute 6D pose in the form (x,y,z,roll,pitch,...
goldfinch's user avatar
0 votes
1 answer
858 views

Is it a bad practice to use cumulative rewards in reinforcement learning

I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over ...
Leibniz's user avatar
  • 69
1 vote
0 answers
120 views

Could we add clipping in the output layer of the actor in DDPG?

I have a doubt about how clipping affects the training of the RL agents. In particular, I have come across a code for training DDPG agents, the pseudo-code is the following: ...
Leibniz's user avatar
  • 69
1 vote
0 answers
58 views

Setting initial values in DDPG to favor better actions

I'm working on a problem using DDPG. Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
jazz's user avatar
  • 11
1 vote
0 answers
242 views

How to parallelize multi-agent DDPG (MADDPG)

I am experimenting with MADDPG algorithm implemented in this repo. Since there were only a few agents (2-3) in the implementation (also in the original paper) steps like parameter updates, action ...
Mika's user avatar
  • 341
0 votes
1 answer
205 views

Is it possible to solve a linear programming problem using reinforcement learning? (DDPG algorithm)

I'm trying to solve a linear programming problem using reinforcement learning. The linear programming problem is: \begin{array}{ll} \text{maximize}_x & C* x \\ \text{subject to}& A*x \le b\\ ...
Avarash Goel's user avatar
3 votes
0 answers
359 views

How to deal with a moving target in the Lunar Lander environment with DDPG?

I have noticed that DDPG does rather well at solving environments with a static target. For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
user1779362's user avatar
1 vote
1 answer
26 views

How to have zero value or a value between 200 and 400 in the output of a deep learning model?

I want to implement a DDPG method and obviously, the action space will be continuous. I have three outputs. The first output should be zero or a value between 200 and 400, and the other outputs have ...
Jacksss's user avatar
  • 11
1 vote
1 answer
185 views

Why is the behaviour policy denoted by $\beta$ and the exploration policy by $ \mu'$ in the DDPG paper?

I am learning about the deep deterministic policy gradient (DDPG) (Lillicrap et al, 2016) and got confused about the notation of the behavior policy. Lillicrap et al. denote the policy gradient by $$\...
Manuel's user avatar
  • 45
2 votes
1 answer
336 views

Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after ...
Dheerendra Singh Tomar's user avatar
1 vote
0 answers
201 views

Gradual decrease in performance of a DDPG agent

I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some ...
Hirek Kubica's user avatar
2 votes
1 answer
264 views

How does DDPG algorithm know about my action mapping function?

I am using DDPG to solve a RL problem. The action space is given by the Cartesian product $[0,20]^4\times[0,6]^4$. The actor is implemented as a deep neural network ...
zdm's user avatar
  • 301
1 vote
0 answers
298 views

How to deal with KerasRL DDPG algorithm getting stuck in a local optima?

I am using KerasRL DDPG to try to learn a policy on my own custom environment, but the agent is stuck in a local optima although I am adding the OrnsteinUhlenbeck randomization process. I used the ...
BAKYAC's user avatar
  • 36
1 vote
0 answers
190 views

Why would DDPG with Hindsight Experience Replay not converge?

I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
Vedant Shah's user avatar
5 votes
1 answer
3k views

How does the Ornstein-Uhlenbeck process work, and how it is used in DDPG?

In section 3 of the paper Continuous control with deep reinforcement learning, the authors write As detailed in the supplementary materials we used an Ornstein-Uhlenbeck process (Uhlenbeck & ...
dani's user avatar
  • 51
1 vote
1 answer
91 views

What do the state features of KukaGymEnv represent? [closed]

I trying to use DDPG augmented with Hindsight Experience Replay (HER) on pybullet's KukaGymEnv. To formulate the feature vector for the goal state, I need to know what the features of the state of the ...
Vedant Shah's user avatar
1 vote
1 answer
2k views

Using DDPG for control in multi-dimensional continuous action space?

I am relatively new to reinforcement learning, and I am trying to implement a reinforcement learning algorithm that can do continuous control in a custom environment. The state of the environment is ...
Dale Larie's user avatar
1 vote
1 answer
2k views

DDPG doesn't converge for MountainCarContinuous-v0 gym environment

I am trying to implement Deep Deterministic policy gradient algorithm by referring to the paper Continuous Control using Deep Reinforcement Learning on the MountainCarContinuous-v0 gym environment. I ...
Vedant Shah's user avatar
0 votes
0 answers
722 views

Which is the best RL algo for continuous states but discrete action spaces problem

I am trying to train an AI with an environment where the states are continuous but the actions are discrete, that means I can not apply DDPG or TD3. Can someone please help to let know what should be ...
user2783767's user avatar
2 votes
0 answers
944 views

Why is DDPG not learning and it does not converge?

I have used a different setting, but DDPG is not learning and it does not converge. I have used these codes 1,2, and 3 and I used different optimizers, activation functions, and learning rate but ...
I_Al-thamary's user avatar
2 votes
1 answer
543 views

Why is the policy loss the mean of $-Q(s, \mu(s))$ in the DDPG algorithm?

I am trying to implement the DDPG algorithm based on this paper. The part that confuses me is the actor network's update. I don't understand why the policy loss is simply the mean of $-Q(s, \mu(s))$, ...
Dhanush Giriyan's user avatar
1 vote
1 answer
43 views

Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?

I'm looking at some baseline implementations of RL agents on the Pendulum environment. My guess was to use a relatively small neural net (~100 parameters). I'm comparing my solution with some ...
Kris's user avatar
  • 171
2 votes
1 answer
326 views

Why does this Keras implementation of the DDPG algorithm update the critic's network using the gradient but the pseudocode doesn't?

I'm trying to understand the DDPG algorithm using Keras I found the site and started analyzing the code, I can't understand 2 things. The algorithm used to write the code presented on the page In the ...
Hubert's user avatar
  • 21
0 votes
1 answer
103 views

What reinforcement learning algorithm should I use in continuous states?

I want to use reinforcement learning in an environment I made. The exact environment doesn't really matter, but it comes down to this: The amount of different states in the environment is infinite e.g....
SirPVP's user avatar
  • 3
2 votes
1 answer
64 views

Is there a good website where I can learn about Deep Deterministic Policy Gradient?

Is there a good website where I can learn about Deep Deterministic Policy Gradient?
Huzaifah Shamim's user avatar
1 vote
1 answer
205 views

In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?

I'm trying to understand the DDPG algorithm shown at this page. I don't know what should the result of the gradient at step 14 be. Is it a scalar that I have to use to update all the weights (so all ...
unter_983's user avatar
  • 331
4 votes
1 answer
1k views

What made your DDPG implementation on your environment work?

I am working on scheduling problem that has inherent randomness. The dimensions of action and state spaces are 1 and 5 respectively. I am using DDPG, but it seems extremely unstable, and so far it ...
Schach21's user avatar
  • 242
1 vote
0 answers
87 views

Benchmarking SAC on Pybullet

So far I have seen TD3 and DDPG benchmarks on Pybullet environments, but I am looking for SAC benchmarks on Pybullet too, anyone can help?
ASA's user avatar
  • 151
2 votes
0 answers
328 views

A question about the Wolpertinger algorithm (Deep RL in Large Discrete Action Spaces paper)

I am trying to reproduce the recommender task experiment from this paper. The paper suggests to embed discrete actions into continuous action space and then to use the proposed Wolpertinger agent. ...
Pavel Korobov's user avatar
2 votes
0 answers
213 views

How can DDPG handle the discrete action space?

I am wondering how can DDPG or DPG handle the discrete action space. There are some papers saying that use Gumbel softmax with DDPG can make the discrete action problem be solved. However, will the ...
Jarvis's user avatar
  • 41
1 vote
0 answers
31 views

What if the rewards induced by an environment are related to the policy too?

Assume we have a policy $\pi_{\theta}$ in a classic reinforcement learning setting, and a reward function $R^{\pi}(s,a)$ that changes as long as $\pi$ changes i.e. not only is it predefined by the ...
ASA's user avatar
  • 151
2 votes
2 answers
1k views

Why is DDPG an off-policy RL algorithm?

In DDPG, if there are no $\epsilon$-greedy and no action noise, is DDPG an on-policy algorithm?
GoingMyWay's user avatar
3 votes
1 answer
807 views

Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity

I'm working on a RL problem with the following properties: The rewards are extremely sparse i.e. all rewards are 0 except the terminal non-zero reward. Ideally I would not use any reward engineering ...
BGa's user avatar
  • 229
1 vote
0 answers
56 views

Why does the result when restoring a saved DDPG model differ significantly from the result when saving it?

I save the trained model after a certain number of episodes with the special save() function of the DDPG class (the network is saved when the reward reaches zero), but when I restore the model again ...
Ne1zvestnyj's user avatar
7 votes
2 answers
1k views

Why are reinforcement learning methods sample inefficient?

Reinforcement learning methods are considered to be extremely sample inefficient. For example, in a recent DeepMind paper by Hessel et al., they showed that in order to reach human-level performance ...
rrz0's user avatar
  • 263
1 vote
2 answers
324 views

Continuous control with DDPG: How to eliminate steady state error?

Currently I'm working on a continuous control problem using DDPG as my RL algorithm. All in all, things are working out quite well, but the algorithm does not show any tendencies to eliminate the ...
opt12's user avatar
  • 171
3 votes
0 answers
910 views

How does adding noise to the action in DDPG help in learning?

I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.
Ahmad Fares's user avatar
4 votes
1 answer
436 views

How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?

I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (in $[-1, 1]$). To this end, I use a ...
opt12's user avatar
  • 171
3 votes
1 answer
266 views

Purpose of using actor-critic algorithms under deterministic MDP dynamics?

One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from ...
BGa's user avatar
  • 229
2 votes
0 answers
152 views

How to learn using DDPG in python solely using a timeseries datasets

I have a lengthy timeseries datasets which contains several variables (from sensors etc) to be classified as actions or states. Providing they are successfully done, I want to learn a control policy ...
JianNius's user avatar
  • 131
2 votes
1 answer
138 views

Why feed actions in later layer in Q network?

I read the DDPG paper, in which the authors state that the actions are fed only later to their Q network: Actions were not included until the 2nd hidden layer of Q. (Sec 7, Experiment Details) So ...
stefanbschneider's user avatar