Questions tagged [ddpg]
For questions related to the reinforcement learning algorithm called Deep Deterministic Policy Gradient (DDPG).
60
questions
0
votes
0
answers
52
views
Unable to interpret DDPG actor-critic loss curves
I am training a DDPG actor-critic agent and ploting rewards and loss curves each episode to track the training evolution. Rewards values in the plot correspond to the total reward per episode divided ...
- 1
0
votes
0
answers
42
views
MADDPG: reward drops fast after updating ac network
I'm using MADDPG to do computation offloading. But the reward drops fast once I start sampling from the replay buffer.
...
- 1
1
vote
0
answers
40
views
Role of $f$ Target Network in DDPG
I am trying to create a variant of DDPG in MATLAB that has no action-value $\langle Q \rangle$ net, but that instead works with networks $\langle V \rangle, \langle f \rangle, \langle r \rangle$, and ...
1
vote
0
answers
348
views
Why Soft Actor-Critic (SAC) uses a double Q trick?
Twin Delayed DDPG (TD3) uses a double Q trick since the policy is deterministic like in DDPG, which is to mitigate the maximum overestimation bias in DDPG. However, in SAC, the policy is stochastic, ...
- 41
1
vote
2
answers
256
views
RL solutions for OpenAI Gym environments?
Is there any place where people share their agent's settings for solving OpenAI Gym Environments?
For example, I'd like to know what are good parameters for a DDPG agent to learn the task in Reacher-...
- 41
1
vote
0
answers
27
views
Why can't I train like a dataset of samples instead of maintaining replay buffer?
On observing the DDPG algorithm, we notice that the updation of neural networks is happening during the episode.
But, it seems there is no issue if we allow the completion of an episode and then treat ...
- 3,612
2
votes
1
answer
96
views
How to sample the tuples during the initial time steps of the DDPG algorithm?
I am facing an issue in understanding the following line from the pseudocode of the DDPG algorithm
Sample a random minibatch of $N$ transitions $(s_i, a_i, r_i, s_{i+1})$ from $R$
Here $N$ is a ...
- 3,612
1
vote
0
answers
109
views
DDPG agent with convolutional layers for feature extraction [closed]
I'm trying to come up with a definition of the critic for a DDPG agent in PyTorch using a CNN as a feature extractor. It is pretty straight forward for the actor model. However, for the critic model I ...
0
votes
1
answer
132
views
Replay buffer action range in DDPG
I have an environment where the agent action is in range [0, 1.57]. My actor network in DDPG has a tanh activation, and so the network values are in the range [-1,1]. Hence I change the scaling from [-...
- 13
0
votes
1
answer
403
views
Why does my actor-critic network always give either -1 or 1 at the output layer?
I have an actor-critic network. The state space contains continuous variables with different ranges like (0,1.57) and (-0.70, 0.70). And it also contain absolute 6D pose in the form (x,y,z,roll,pitch,...
- 13
0
votes
1
answer
858
views
Is it a bad practice to use cumulative rewards in reinforcement learning
I am using a DDPG agent for doing prediction on the position on an asset in a stock trading-like environment. I am using the cumulative reward as the reward for each timestep. Since it is trained over ...
- 69
1
vote
0
answers
120
views
Could we add clipping in the output layer of the actor in DDPG?
I have a doubt about how clipping affects the training of the RL agents.
In particular, I have come across a code for training DDPG agents, the pseudo-code is the following:
...
- 69
1
vote
0
answers
58
views
Setting initial values in DDPG to favor better actions
I'm working on a problem using DDPG.
Is it possible to add some intelligence in the initialization phase, such that the convergence time is improved/shortened and local optima are avoided as much as ...
- 11
1
vote
0
answers
242
views
How to parallelize multi-agent DDPG (MADDPG)
I am experimenting with MADDPG algorithm implemented in this repo. Since there were only a few agents (2-3) in the implementation (also in the original paper) steps like parameter updates, action ...
- 341
0
votes
1
answer
205
views
Is it possible to solve a linear programming problem using reinforcement learning? (DDPG algorithm)
I'm trying to solve a linear programming problem using reinforcement learning.
The linear programming problem is:
\begin{array}{ll}
\text{maximize}_x & C* x \\
\text{subject to}& A*x \le b\\
...
3
votes
0
answers
359
views
How to deal with a moving target in the Lunar Lander environment with DDPG?
I have noticed that DDPG does rather well at solving environments with a static target.
For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
- 131
1
vote
1
answer
26
views
How to have zero value or a value between 200 and 400 in the output of a deep learning model?
I want to implement a DDPG method and obviously, the action space will be continuous. I have three outputs. The first output should be zero or a value between 200 and 400, and the other outputs have ...
- 11
1
vote
1
answer
185
views
Why is the behaviour policy denoted by $\beta$ and the exploration policy by $ \mu'$ in the DDPG paper?
I am learning about the deep deterministic policy gradient (DDPG) (Lillicrap et al, 2016) and got confused about the notation of the behavior policy.
Lillicrap et al. denote the policy gradient by
$$\...
- 45
2
votes
1
answer
336
views
Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?
I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after ...
1
vote
0
answers
201
views
Gradual decrease in performance of a DDPG agent
I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some ...
- 11
2
votes
1
answer
264
views
How does DDPG algorithm know about my action mapping function?
I am using DDPG to solve a RL problem. The action space is given by the Cartesian product $[0,20]^4\times[0,6]^4$. The actor is implemented as a deep neural network ...
- 301
1
vote
0
answers
298
views
How to deal with KerasRL DDPG algorithm getting stuck in a local optima?
I am using KerasRL DDPG to try to learn a policy on my own custom environment, but the agent is stuck in a local optima although I am adding the OrnsteinUhlenbeck randomization process. I used the ...
- 36
1
vote
0
answers
190
views
Why would DDPG with Hindsight Experience Replay not converge?
I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
- 125
5
votes
1
answer
3k
views
How does the Ornstein-Uhlenbeck process work, and how it is used in DDPG?
In section 3 of the paper Continuous control with deep reinforcement learning, the authors write
As detailed in the supplementary materials we used an Ornstein-Uhlenbeck process (Uhlenbeck & ...
- 51
1
vote
1
answer
91
views
What do the state features of KukaGymEnv represent? [closed]
I trying to use DDPG augmented with Hindsight Experience Replay (HER) on pybullet's KukaGymEnv.
To formulate the feature vector for the goal state, I need to know what the features of the state of the ...
- 125
1
vote
1
answer
2k
views
Using DDPG for control in multi-dimensional continuous action space?
I am relatively new to reinforcement learning, and I am trying to implement a reinforcement learning algorithm that can do continuous control in a custom environment. The state of the environment is ...
- 23
1
vote
1
answer
2k
views
DDPG doesn't converge for MountainCarContinuous-v0 gym environment
I am trying to implement Deep Deterministic policy gradient algorithm by referring to the paper Continuous Control using Deep Reinforcement Learning on the MountainCarContinuous-v0 gym environment. I ...
- 125
0
votes
0
answers
722
views
Which is the best RL algo for continuous states but discrete action spaces problem
I am trying to train an AI with an environment where the states are continuous but the actions are discrete, that means I can not apply DDPG or TD3.
Can someone please help to let know what should be ...
- 121
2
votes
0
answers
944
views
Why is DDPG not learning and it does not converge?
I have used a different setting, but DDPG is not learning and it does not converge. I have used these codes 1,2, and 3 and I used different optimizers, activation functions, and learning rate but ...
- 52
2
votes
1
answer
543
views
Why is the policy loss the mean of $-Q(s, \mu(s))$ in the DDPG algorithm?
I am trying to implement the DDPG algorithm based on this paper.
The part that confuses me is the actor network's update.
I don't understand why the policy loss is simply the mean of $-Q(s, \mu(s))$, ...
- 345
1
vote
1
answer
43
views
Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?
I'm looking at some baseline implementations of RL agents on the Pendulum environment. My guess was to use a relatively small neural net (~100 parameters).
I'm comparing my solution with some ...
- 171
2
votes
1
answer
326
views
Why does this Keras implementation of the DDPG algorithm update the critic's network using the gradient but the pseudocode doesn't?
I'm trying to understand the DDPG algorithm using Keras
I found the site and started analyzing the code, I can't understand 2 things.
The algorithm used to write the code presented on the page
In the ...
- 21
0
votes
1
answer
103
views
What reinforcement learning algorithm should I use in continuous states?
I want to use reinforcement learning in an environment I made. The exact environment doesn't really matter, but it comes down to this: The amount of different states in the environment is infinite e.g....
- 3
2
votes
1
answer
64
views
Is there a good website where I can learn about Deep Deterministic Policy Gradient?
Is there a good website where I can learn about Deep Deterministic Policy Gradient?
1
vote
1
answer
205
views
In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?
I'm trying to understand the DDPG algorithm shown at this page. I don't know what should the result of the gradient at step 14 be.
Is it a scalar that I have to use to update all the weights (so all ...
- 331
4
votes
1
answer
1k
views
What made your DDPG implementation on your environment work?
I am working on scheduling problem that has inherent randomness. The dimensions of action and state spaces are 1 and 5 respectively.
I am using DDPG, but it seems extremely unstable, and so far it ...
- 242
1
vote
0
answers
87
views
Benchmarking SAC on Pybullet
So far I have seen TD3 and DDPG benchmarks on Pybullet environments, but I am looking for SAC benchmarks on Pybullet too, anyone can help?
- 151
2
votes
0
answers
328
views
A question about the Wolpertinger algorithm (Deep RL in Large Discrete Action Spaces paper)
I am trying to reproduce the recommender task experiment from this paper.
The paper suggests to embed discrete actions into continuous action space and then to use the proposed Wolpertinger agent.
...
2
votes
0
answers
213
views
How can DDPG handle the discrete action space?
I am wondering how can DDPG or DPG handle the discrete action space. There are some papers saying that use Gumbel softmax with DDPG can make the discrete action problem be solved. However, will the ...
- 41
1
vote
0
answers
31
views
What if the rewards induced by an environment are related to the policy too?
Assume we have a policy $\pi_{\theta}$ in a classic reinforcement learning setting, and a reward function $R^{\pi}(s,a)$ that changes as long as $\pi$ changes i.e. not only is it predefined by the ...
- 151
2
votes
2
answers
1k
views
Why is DDPG an off-policy RL algorithm?
In DDPG, if there are no $\epsilon$-greedy and no action noise, is DDPG an on-policy algorithm?
- 150
3
votes
1
answer
807
views
Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity
I'm working on a RL problem with the following properties:
The rewards are extremely sparse i.e. all rewards are 0 except the terminal non-zero reward. Ideally I would not use any reward engineering ...
- 229
1
vote
0
answers
56
views
Why does the result when restoring a saved DDPG model differ significantly from the result when saving it?
I save the trained model after a certain number of episodes with the special save() function of the DDPG class (the network is saved when the reward reaches zero), but when I restore the model again ...
- 111
7
votes
2
answers
1k
views
Why are reinforcement learning methods sample inefficient?
Reinforcement learning methods are considered to be extremely sample inefficient.
For example, in a recent DeepMind paper by Hessel et al., they showed that in order to reach human-level performance ...
- 263
1
vote
2
answers
324
views
Continuous control with DDPG: How to eliminate steady state error?
Currently I'm working on a continuous control problem using DDPG as my RL algorithm. All in all, things are working out quite well, but the algorithm does not show any tendencies to eliminate the ...
- 171
3
votes
0
answers
910
views
How does adding noise to the action in DDPG help in learning?
I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.
- 31
4
votes
1
answer
436
views
How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?
I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (in $[-1, 1]$).
To this end, I use a ...
- 171
3
votes
1
answer
266
views
Purpose of using actor-critic algorithms under deterministic MDP dynamics?
One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from ...
- 229
2
votes
0
answers
152
views
How to learn using DDPG in python solely using a timeseries datasets
I have a lengthy timeseries datasets which contains several variables (from sensors etc) to be classified as actions or states. Providing they are successfully done, I want to learn a control policy ...
- 131
2
votes
1
answer
138
views
Why feed actions in later layer in Q network?
I read the DDPG paper, in which the authors state that the actions are fed only later to their Q network:
Actions were not included until the 2nd hidden layer of Q. (Sec 7, Experiment Details)
So ...
- 183