People who code: we want your input. Take the Survey

Questions tagged [ddpg]

For questions related to the reinforcement learning algorithm called Deep Deterministic Policy Gradient (DDPG).

29 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7
votes
0answers
814 views

Is there a difference in the architecture of deep reinforcement learning when multiple actions are performed instead of a single action?

I've built a deep deterministic policy gradient reinforcement learning agent to be able to handle any games/tasks that have only one action. However, the agent seems to fail horribly when there are ...
4
votes
0answers
282 views

Policy gradient methods for continuous action space

I have a problem I would like to tackle with RL but I am not sure if it is even doable. My agent has to figure out how to fill a very large vector (let's say from 600 to 4000 in the most complex ...
4
votes
1answer
129 views

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

Twin Delayed Deep Deterministic (TD3) policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained ...
3
votes
0answers
90 views

How to deal with a moving target in the Lunar Lander environment with DDPG?

I have noticed that DDPG does rather well at solving environments with a static target. For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
3
votes
0answers
193 views

Should noise (such as OU) be decreased over time in actor / critic algorithms?

In most of RL algorithms I saw, there is a coefficient that reduces actions exploration over time, to help convergence. But in Actor-Critic, or other algorithms (A3C, DDPG, ...) used in continuous ...
3
votes
0answers
103 views

Can I use deterministic policy gradient methods for stochastic policy learning?

Can I treat a stochastic policy (over a finite action space of size $n$) as a deterministic policy (in the set of probability distribution in $\mathbb{R}^n$)? It seems to me that nothing is broken ...
2
votes
0answers
367 views

Why is DDPG not learning and it does not converge?

I have used a different setting, but DDPG is not learning and it does not converge. I have used these codes 1,2, and 3 and I used different optimizers, activation functions, and learning rate but ...
2
votes
0answers
148 views

How does adding noise to the action in DDPG help in learning?

I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.
2
votes
0answers
39 views

How to learn using DDPG in python solely using a timeseries datasets

I have a lengthy timeseries datasets which contains several variables (from sensors etc) to be classified as actions or states. Providing they are successfully done, I want to learn a control policy ...
1
vote
0answers
25 views

Gradual decrease in performance of a DDPG agent

I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some ...
1
vote
0answers
84 views

How to deal with KerasRL DDPG algorithm getting stuck in a local optima?

I am using KerasRL DDPG to try to learn a policy on my own custom environment, but the agent is stuck in a local optima although I am adding the OrnsteinUhlenbeck randomization process. I used the ...
1
vote
0answers
56 views

Why would DDPG with Hindsight Experience Replay not converge?

I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
1
vote
0answers
55 views

Benchmarking SAC on Pybullet

So far I have seen TD3 and DDPG benchmarks on Pybullet environments, but I am looking for SAC benchmarks on Pybullet too, anyone can help?
1
vote
0answers
135 views

A question about the Wolpertinger algorithm (Deep RL in Large Discrete Action Spaces paper)

I am trying to reproduce the recommender task experiment from this paper. The paper suggests to embed discrete actions into continuous action space and then to use the proposed Wolpertinger agent. ...
1
vote
0answers
67 views

How can DDPG handle the discrete action space?

I am wondering how can DDPG or DPG handle the discrete action space. There are some papers saying that use Gumbel softmax with DDPG can make the discrete action problem be solved. However, will the ...
1
vote
0answers
26 views

What if the rewards induced by an environment are related to the policy too?

Assume we have a policy $\pi_{\theta}$ in a classic reinforcement learning setting, and a reward function $R^{\pi}(s,a)$ that changes as long as $\pi$ changes i.e. not only is it predefined by the ...
1
vote
0answers
32 views

Why does the result when restoring a saved DDPG model differ significantly from the result when saving it?

I save the trained model after a certain number of episodes with the special save() function of the DDPG class (the network is saved when the reward reaches zero), but when I restore the model again ...
1
vote
0answers
24 views

DDPG: how to implement continuous action space bounded in the interval [-2, 2]?

I am a newbie in reinforcement learning and trying to understand how to implement continuous actions bounded by $[-2, 2]$. My research shows that doing nothing is a possible solution (i.e. action of 4....
1
vote
0answers
30 views

Should we multiply the target of actor by the importance sampling ratio when prioritized replay is applied to DDPG?

According to PER, we have to multiply the $Q$ error $\delta_i$ by the importance sampling ratio to correct the bias introduced by the imbalance sampling of PER, where importance sampling ratio is ...
0
votes
0answers
25 views

What is the relationship between buffer size and actor loss in DDPG?

Actor loss in the DDPG algorithm is: critic_value = critic_model([state_batch, action_batch]) actor_loss = -tf.math.reduce_mean(critic_value) As I was trying to ...
0
votes
0answers
36 views

How to scale the action in a custom environment with DDPG?

I am trying to implement DDPG in a custom gym environment. The action is the relative allocation of funds between each asset. The action space is a Box with the ...
0
votes
0answers
17 views

Which loss function should I use to train DDGP with multiple q values, one for each of the output dimensions?

I'm trying to come up with a loss function for the case, in DDPG, where we have as many outputs from the critic as there are from the actor. So, there will be one Q value for each dimension in the ...
0
votes
0answers
50 views

Does Multi-Agent Deep Deterministic Policy Gradient also work with discrete action spaces?

I would like to ask if Multi-Agent Deep Deterministic Policy Gradient (MADDPG) works fine with discrete action space. DDPG works only with continuous action space, but I have read that MADDPG can also ...
0
votes
0answers
29 views

Why is actor's loss the mean of the predicted Q values and not an error in calculating Q values, in the DDPG algorithm?

I am trying to implement the DDPG algorithm. Why is the actor's loss calculated as the negative mean of the model's predicted Q values in the states we are in? Shouldn't it be like the difference of ...
0
votes
0answers
97 views

Which is the best RL algo for continuous states but discrete action spaces problem

I am trying to train an AI with an environment where the states are continuous but the actions are discrete, that means I can not apply DDPG or TD3. Can someone please help to let know what should be ...
0
votes
0answers
84 views

Why is my DDPG agent (implemented in TensorFlow) not learning?

I am trying to implement a Reinforcement Learning algorithm called DDPG in TensorFlow 2.x on a custom gym environment. I am new to TF. So, I started with the DDPG TF 1.x implementation from pemami4911....
0
votes
0answers
35 views

2 out of 7 Observations Defined in MATLAB DDPG Reinforcement Learning Environment. Are the rest given random values?

After reading up one Deep Deterministic Policy Gradient, I found this example on MATLAB: https://www.mathworks.com/help/reinforcement-learning/ug/train-agent-to-control-flying-robot.html#...
0
votes
1answer
80 views

In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?

I'm trying to understand the DDPG algorithm shown at this page. I don't know what should the result of the gradient at step 14 be. Is it a scalar that I have to use to update all the weights (so all ...
0
votes
0answers
111 views

Using deep deterministic policy gradient in OpenAI Gym to solve problems with continuous actions

I am trying to do the following: Install the OpenAI baseline algorithms from the following GitHub repository: github.com/openai/baselines by following the instructions in the readme file. Train an ...