Questions tagged [ddpg]

For questions related to the reinforcement learning algorithm called Deep Deterministic Policy Gradient (DDPG).

Filter by
Sorted by
Tagged with
1
vote
0answers
29 views

Could we add clipping in the output layer of the actor in DDPG?

I have a doubt about how clipping affects the training of the RL agents. In particular, I have come across a code for training DDPG agents, the pseudo-code is the following: ...
1
vote
0answers
31 views

Setting initial values in DDPG to favor better actions

I'm working on a problem using DDPG. Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
1
vote
0answers
26 views

How to parallelize multi-agent DDPG (MADDPG)

I am experimenting with MADDPG algorithm implemented in this repo. Since there were only a few agents (2-3) in the implementation (also in the original paper) steps like parameter updates, action ...
0
votes
1answer
75 views

Is it possible to solve a linear programming problem using reinforcement learning? (DDPG algorithm)

I'm trying to solve a linear programming problem using reinforcement learning. The linear programming problem is: \begin{array}{ll} \text{maximize}_x & C* x \\ \text{subject to}& A*x \le b\\ ...
0
votes
0answers
18 views

DDPG or PPO don't work well with my custom non gym environment

I have a project to control a robot with right and left wheel speeds, and my step time is not constant. Because my outputs are continuous (right wheel speed, left wheel speed, and time step), I try to ...
0
votes
0answers
34 views

What is the relationship between buffer size and actor loss in DDPG?

Actor loss in the DDPG algorithm is: critic_value = critic_model([state_batch, action_batch]) actor_loss = -tf.math.reduce_mean(critic_value) As I was trying to ...
3
votes
0answers
102 views

How to deal with a moving target in the Lunar Lander environment with DDPG?

I have noticed that DDPG does rather well at solving environments with a static target. For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
1
vote
1answer
20 views

How to have zero value or a value between 200 and 400 in the output of a deep learning model?

I want to implement a DDPG method and obviously, the action space will be continuous. I have three outputs. The first output should be zero or a value between 200 and 400, and the other outputs have ...
0
votes
0answers
81 views

How to scale the action in a custom environment with DDPG?

I am trying to implement DDPG in a custom gym environment. The action is the relative allocation of funds between each asset. The action space is a Box with the ...
1
vote
1answer
45 views

Why is the behaviour policy denoted by $\beta$ and the exploration policy by $ \mu'$ in the DDPG paper?

I am learning about the deep deterministic policy gradient (DDPG) (Lillicrap et al, 2016) and got confused about the notation of the behavior policy. Lillicrap et al. denote the policy gradient by $$\...
2
votes
1answer
91 views

Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after ...
1
vote
0answers
37 views

Gradual decrease in performance of a DDPG agent

I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some ...
0
votes
0answers
19 views

Which loss function should I use to train DDGP with multiple q values, one for each of the output dimensions?

I'm trying to come up with a loss function for the case, in DDPG, where we have as many outputs from the critic as there are from the actor. So, there will be one Q value for each dimension in the ...
2
votes
1answer
37 views

How does DDPG algorithm know about my action mapping function?

I am using DDPG to solve a RL problem. The action space is given by the Cartesian product $[0,20]^4\times[0,6]^4$. The actor is implemented as a deep neural network ...
1
vote
0answers
102 views

How to deal with KerasRL DDPG algorithm getting stuck in a local optima?

I am using KerasRL DDPG to try to learn a policy on my own custom environment, but the agent is stuck in a local optima although I am adding the OrnsteinUhlenbeck randomization process. I used the ...
1
vote
0answers
64 views

Why would DDPG with Hindsight Experience Replay not converge?

I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
4
votes
1answer
1k views

How does the Ornstein-Uhlenbeck process work, and how it is used in DDPG?

In section 3 of the paper Continuous control with deep reinforcement learning, the authors write As detailed in the supplementary materials we used an Ornstein-Uhlenbeck process (Uhlenbeck & ...
1
vote
1answer
69 views

What do the state features of KukaGymEnv represent?

I trying to use DDPG augmented with Hindsight Experience Replay (HER) on pybullet's KukaGymEnv. To formulate the feature vector for the goal state, I need to know what the features of the state of the ...
1
vote
1answer
355 views

Using DDPG for control in multi-dimensional continuous action space?

I am relatively new to reinforcement learning, and I am trying to implement a reinforcement learning algorithm that can do continuous control in a custom environment. The state of the environment is ...
1
vote
1answer
445 views

DDPG doesn't converge for MountainCarContinuous-v0 gym environment

I am trying to implement Deep Deterministic policy gradient algorithm by referring to the paper Continuous Control using Deep Reinforcement Learning on the MountainCarContinuous-v0 gym environment. I ...
0
votes
0answers
160 views

Which is the best RL algo for continuous states but discrete action spaces problem

I am trying to train an AI with an environment where the states are continuous but the actions are discrete, that means I can not apply DDPG or TD3. Can someone please help to let know what should be ...
2
votes
0answers
426 views

Why is DDPG not learning and it does not converge?

I have used a different setting, but DDPG is not learning and it does not converge. I have used these codes 1,2, and 3 and I used different optimizers, activation functions, and learning rate but ...
1
vote
1answer
73 views

Why is the policy loss the mean of $-Q(s, \mu(s))$ in the DDPG algorithm?

I am trying to implement the DDPG algorithm based on this paper. The part that confuses me is the actor network's update. I don't understand why the policy loss is simply the mean of $-Q(s, \mu(s))$, ...
1
vote
1answer
39 views

Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?

I'm looking at some baseline implementations of RL agents on the Pendulum environment. My guess was to use a relatively small neural net (~100 parameters). I'm comparing my solution with some ...
2
votes
1answer
180 views

Understand the DDPG algorithm in Keras

I'm trying to understand the DDPG algorithm using Keras I found the site and started analyzing the code, I can't understand 2 things. The algorithm used to write the code presented on the page In the ...
0
votes
1answer
54 views

What reinforcement learning algorithm should I use in continuous states?

I want to use reinforcement learning in an environment I made. The exact environment doesn't really matter, but it comes down to this: The amount of different states in the environment is infinite e.g....
2
votes
1answer
56 views

Is there a good website where I can learn about Deep Deterministic Policy Gradient?

Is there a good website where I can learn about Deep Deterministic Policy Gradient?
0
votes
1answer
90 views

In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?

I'm trying to understand the DDPG algorithm shown at this page. I don't know what should the result of the gradient at step 14 be. Is it a scalar that I have to use to update all the weights (so all ...
3
votes
1answer
325 views

What made your DDPG implementation on your environment work?

I am working on scheduling problem that has inherent randomness. The dimensions of action and state spaces are 1 and 5 respectively. I am using DDPG, but it seems extremely unstable, and so far it ...
1
vote
0answers
58 views

Benchmarking SAC on Pybullet

So far I have seen TD3 and DDPG benchmarks on Pybullet environments, but I am looking for SAC benchmarks on Pybullet too, anyone can help?
1
vote
0answers
157 views

A question about the Wolpertinger algorithm (Deep RL in Large Discrete Action Spaces paper)

I am trying to reproduce the recommender task experiment from this paper. The paper suggests to embed discrete actions into continuous action space and then to use the proposed Wolpertinger agent. ...
1
vote
0answers
83 views

How can DDPG handle the discrete action space?

I am wondering how can DDPG or DPG handle the discrete action space. There are some papers saying that use Gumbel softmax with DDPG can make the discrete action problem be solved. However, will the ...
1
vote
0answers
26 views

What if the rewards induced by an environment are related to the policy too?

Assume we have a policy $\pi_{\theta}$ in a classic reinforcement learning setting, and a reward function $R^{\pi}(s,a)$ that changes as long as $\pi$ changes i.e. not only is it predefined by the ...
2
votes
2answers
424 views

Why is DDPG an off-policy RL algorithm?

In DDPG, if there are no $\epsilon$-greedy and no action noise, is DDPG an on-policy algorithm?
3
votes
1answer
212 views

Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity

I'm working on a RL problem with the following properties: The rewards are extremely sparse i.e. all rewards are 0 except the terminal non-zero reward. Ideally I would not use any reward engineering ...
1
vote
0answers
34 views

Why does the result when restoring a saved DDPG model differ significantly from the result when saving it?

I save the trained model after a certain number of episodes with the special save() function of the DDPG class (the network is saved when the reward reaches zero), but when I restore the model again ...
6
votes
2answers
369 views

Why are reinforcement learning methods sample inefficient?

Reinforcement learning methods are considered to be extremely sample inefficient. For example, in a recent DeepMind paper by Hessel et al., they showed that in order to reach human-level performance ...
1
vote
2answers
117 views

Continuous control with DDPG: How to eliminate steady state error?

Currently I'm working on a continuous control problem using DDPG as my RL algorithm. All in all, things are working out quite well, but the algorithm does not show any tendencies to eliminate the ...
2
votes
0answers
203 views

How does adding noise to the action in DDPG help in learning?

I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.
3
votes
1answer
166 views

How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?

I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (in $[-1, 1]$). To this end, I use a ...
3
votes
1answer
117 views

Purpose of using actor-critic algorithms under deterministic MDP dynamics?

One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from ...
2
votes
0answers
43 views

How to learn using DDPG in python solely using a timeseries datasets

I have a lengthy timeseries datasets which contains several variables (from sensors etc) to be classified as actions or states. Providing they are successfully done, I want to learn a control policy ...
2
votes
1answer
68 views

Why feed actions in later layer in Q network?

I read the DDPG paper, in which the authors state that the actions are fed only later to their Q network: Actions were not included until the 2nd hidden layer of Q. (Sec 7, Experiment Details) So ...
1
vote
0answers
29 views

DDPG: how to implement continuous action space bounded in the interval [-2, 2]?

I am a newbie in reinforcement learning and trying to understand how to implement continuous actions bounded by $[-2, 2]$. My research shows that doing nothing is a possible solution (i.e. action of 4....
4
votes
0answers
293 views

Policy gradient methods for continuous action space

I have a problem I would like to tackle with RL but I am not sure if it is even doable. My agent has to figure out how to fill a very large vector (let's say from 600 to 4000 in the most complex ...
0
votes
1answer
268 views

Is DDPG just for deterministic environments?

I want to develop an AI for continuous space. I reached to DDPG algorithm that takes actions deterministically. If DDPG takes actions deterministically, should the environment also be deterministic? ...
3
votes
1answer
114 views

Training actor-critic algorithms in games with opponents

I am wondering how am I supposed to train a model using actor/critic algorithms in environments with opponents. I tried the followings (using A3C and DDPG): Play against random player. I had rather ...
3
votes
0answers
228 views

Should noise (such as OU) be decreased over time in actor / critic algorithms?

In most of RL algorithms I saw, there is a coefficient that reduces actions exploration over time, to help convergence. But in Actor-Critic, or other algorithms (A3C, DDPG, ...) used in continuous ...
7
votes
2answers
6k views

How large should the replay buffer be?

I'm learning DDPG algorithm by following the following link: Open AI Spinning Up document on DDPG, where it is written In order for the algorithm to have stable behavior, the replay buffer should ...
4
votes
1answer
138 views

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

Twin Delayed Deep Deterministic (TD3) policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained ...