Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [double-dqn]

For questions about the double DQN model introduced in the paper "Deep Reinforcement Learning with Double Q-learning" (2015) by Hado van Hasselt et al.

Filter by
Sorted by
Tagged with
0 votes
1 answer
33 views

How Come My (D)DQN Fails To Learn?

I am currently trying to teach a (D)DQN algorithm to play a 10x10 GridWorld game, so I can compare the two as I increase the number of moves the agent can take. The rewards are as follows: A step = -1 ...
user avatar
0 votes
1 answer
34 views

Does "number of actions" refer to the number of actions taken or size of the action space?

In the original DDQN article (https://arxiv.org/pdf/1509.06461.pdf,) the phrase "number of actions" is used twice; First, in the following context: Secondly in Theorem 1. I have a hard ...
user avatar
0 votes
0 answers
35 views

How Does The DDQN Step/interact In The Environment?

I have made a (D)DQN Model. In this model, regardless of whether I initialize it in DDQN or DQN mode, it uses an experience replay memory. The way I gather transitions for this experience replay ...
user avatar
0 votes
0 answers
40 views

Should you clip Q values if they start to grow indefinitely?

I am training the SAC algorithm for an environment where the rewards are small as shown below and the episode length is 84. I have a problem with the Q values that grow with each step. The following ...
user avatar
  • 167
0 votes
0 answers
28 views

dDQN converges, but makes suboptimal decisions that are mainly focused on retrieving short-term reward

I use a dDQN to dispatch drivers in a ride-hailing environment. The action space has size (#drivers + 1), which means we can choose one of the drivers or choose to refuse an order (and wait for the ...
user avatar
  • 3
2 votes
0 answers
202 views

Update Rule with Deep Q-Learning (DQN) for 2-player games

I am wondering how to correctly implement the DQN algorithm for two-player games such as Tic Tac Toe and Connect 4. While my algorithm is mastering Tic Tac Toe relatively quickly, I cannot get great ...
user avatar
  • 31
0 votes
0 answers
68 views

How should I model the state and action spaces for a problem where the goal is to draw a line between two points?

I have a problem where the goal is for the agent to draw a single line between two points on a $500 \times 500$ white image. I have built my DQN. For now, the output layer's size of the network is $[...
user avatar
  • 323
4 votes
1 answer
663 views

Why does regular Q-learning (and DQN) overestimate the Q values?

The motivation for the introduction of double DQN (and double Q-learning) is that the regular Q-learning (or DQN) can overestimate the Q value, but is there a brief explanation as to why it is ...
user avatar
2 votes
0 answers
21 views

Can DQN outperform DoubleDQN?

I found a similar post about this issue, but unfortunately I did not find a proper answer. Are there any references where DQN is better than DoubleDQN, that is DoubleDQN does not improve DQN ?
user avatar
  • 889
1 vote
0 answers
80 views

DDQN Agent in Othello (Reversi) game struggle to learn

This is my first question on this forum and I would like to welcome everyone. I am trying to implement DDQN Agent playing Othello (Reversi) game. I have tried multiple things but the agent which seems ...
user avatar
1 vote
0 answers
17 views

Would it make sense to share the layers (except the last one) of the neural networks in Double DQN?

Context: Double Q-learning was introduced to prevent the maximization bias from q-learning. Instead of learning a single Q-network, we can learn two (or in general $K > 1$) and our Q-estimate would ...
user avatar
  • 131
0 votes
1 answer
51 views

DQN rgb input channels problem using pytorch

I've been trying to learn about CNN's and reinforcement learning and I found this project to play with: https://github.com/adityajn105/flappy-bird-deep-q-learning I've been trying to change the code ...
user avatar
3 votes
1 answer
102 views

Why do we minimise the loss between the target Q values and 'local' Q values?

I have a question regarding the loss function of target networks and current (online) networks. I understand the action value function. What I am unsure about is why we seek to minimise the loss ...
user avatar
3 votes
1 answer
125 views

How to compute the target for double Q-learning update step?

I've already read the original paper about double DQN but I do not find a clear and practical explanation of how the target $y$ is computed, so here's how I interpreted the method (let's say I have 3 ...
user avatar
  • 281
3 votes
1 answer
757 views

What exactly is the advantage of double DQN over DQN?

I started looking into the double DQN (DDQN). Apparently, the difference between DDQN and DQN is that in DDQN we use the main value network for action selection and the target network for outputting ...
user avatar
  • 239
2 votes
1 answer
184 views

How does the target network in double DQNs find the maximum Q value for each action?

I understand the fact that the neural network is used to take the states as inputs and it outputs the Q-value for state-action pairs. However, in order to compute this and update its weights, we need ...
user avatar
1 vote
1 answer
154 views

Why does adding another network help in double DQN? [duplicate]

What is the idea behind double DQN? The target in double DQN is computed as follows $$ Y_{t}^{\text {DoubleQ }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1},...
user avatar
  • 11
4 votes
1 answer
97 views

What does the notation $p_t = \text{max}_{i<t} p_i$ mean in algorithm 1 of the prioritized experience replay paper?

I am having a hard time converting line 6 of the prioritized experience replay algorithm from the original paper into plain English (see below): I understand that new transitions (not visited before) ...
user avatar
  • 499
7 votes
2 answers
1k views

Can DQN perform better than Double DQN?

I'm training both DQN and double DQN in the same environment, but DQN performs significantly better than double DQN. As I've seen in the double DQN paper, double DQN should perform better than DQN. Am ...
user avatar
  • 171
4 votes
1 answer
2k views

How can I ensure convergence of DDQN, if the true Q-values for different actions in the same state are very close?

I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future ...
user avatar
  • 93