Questions tagged [double-dqn]

For questions about the double DQN model introduced in the paper "Deep Reinforcement Learning with Double Q-learning" (2015) by Hado van Hasselt et al.

Filter by
Sorted by
Tagged with
1 vote
0 answers
62 views

Resulting quantiles from Quantile Regression DQN

In my QR-DQN application, the resulting quantiles for a state s and action a take the form of the blue line in the figure. The ...
Amav's user avatar
  • 11
1 vote
0 answers
27 views

Why slow-changing policy invalidates Double DQN approach in TD3 paper?

In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
Jerry Ding's user avatar
4 votes
1 answer
100 views

Finding the true Q-values in gymnaiusm

I'm very interested in the true Q-values of state-action pairs in the classic control environments in gymnasium. Contrary to the usual goal, the ordering of the Q-values itself is irrelevant; a very ...
Mark B's user avatar
  • 43
0 votes
0 answers
17 views

Training two DQNs at the same time: one for task allocation, the other for action selection

i am currently trying to tackle eternity 2 wich is a tile matching puzzle with a DDQN, but it seems to me that using only one DQN won't do the trick. I am currently using a DQN to select an action ...
mt-clemente's user avatar
1 vote
0 answers
88 views

DDQN Snake keeps crashing into the wall [closed]

Edit: I managed to fix this by changing the optimizer to SGD. I am very new to reinforcement learning, and I attempted to create a DDQN for the game snake but for some reason it keeps learning to ...
ImNotKevPlayz's user avatar
1 vote
0 answers
48 views

What Kind of Reinforcement Learning Algorithms Can Be Used when the Action Space is Unfeasibly Large?

I know Deep Q network as a $S\times A$ DNN which maps the $S$ dimensional statespace to q-values of $A$ distinct actions. In my problem, the action space is still discrete, and finite, but depending ...
Della's user avatar
  • 111
1 vote
1 answer
125 views

How Come My (D)DQN Fails To Learn?

I am currently trying to teach a (D)DQN algorithm to play a 10x10 GridWorld game, so I can compare the two as I increase the number of moves the agent can take. The rewards are as follows: A step = -1 ...
GeorgeWTrump's user avatar
0 votes
1 answer
106 views

Does "number of actions" refer to the number of actions taken or size of the action space?

In the original DDQN article (https://arxiv.org/pdf/1509.06461.pdf,) the phrase "number of actions" is used twice; First, in the following context: Secondly in Theorem 1. I have a hard ...
GeorgeWTrump's user avatar
2 votes
0 answers
396 views

Update Rule with Deep Q-Learning (DQN) for 2-player games

I am wondering how to correctly implement the DQN algorithm for two-player games such as Tic Tac Toe and Connect 4. While my algorithm is mastering Tic Tac Toe relatively quickly, I cannot get great ...
spadel's user avatar
  • 31
0 votes
0 answers
169 views

How should I model the state and action spaces for a problem where the goal is to draw a line between two points?

I have a problem where the goal is for the agent to draw a single line between two points on a $500 \times 500$ white image. I have built my DQN. For now, the output layer's size of the network is $[...
junfanbl's user avatar
  • 323
5 votes
1 answer
2k views

Why does regular Q-learning (and DQN) overestimate the Q values?

The motivation for the introduction of double DQN (and double Q-learning) is that the regular Q-learning (or DQN) can overestimate the Q value, but is there a brief explanation as to why it is ...
ground clown's user avatar
2 votes
0 answers
34 views

Can DQN outperform DoubleDQN?

I found a similar post about this issue, but unfortunately I did not find a proper answer. Are there any references where DQN is better than DoubleDQN, that is DoubleDQN does not improve DQN ?
ddaedalus's user avatar
  • 919
1 vote
0 answers
308 views

DDQN Agent in Othello (Reversi) game struggle to learn

This is my first question on this forum and I would like to welcome everyone. I am trying to implement DDQN Agent playing Othello (Reversi) game. I have tried multiple things but the agent which seems ...
Mikołaj Michalski's user avatar
1 vote
0 answers
20 views

Would it make sense to share the layers (except the last one) of the neural networks in Double DQN?

Context: Double Q-learning was introduced to prevent the maximization bias from q-learning. Instead of learning a single Q-network, we can learn two (or in general $K > 1$) and our Q-estimate would ...
kaiwenw's user avatar
  • 151
0 votes
1 answer
108 views

DQN rgb input channels problem using pytorch

I've been trying to learn about CNN's and reinforcement learning and I found this project to play with: https://github.com/adityajn105/flappy-bird-deep-q-learning I've been trying to change the code ...
aiEnthusiast's user avatar
3 votes
1 answer
268 views

Why do we minimise the loss between the target Q values and 'local' Q values?

I have a question regarding the loss function of target networks and current (online) networks. I understand the action value function. What I am unsure about is why we seek to minimise the loss ...
user9317212's user avatar
3 votes
1 answer
375 views

How to compute the target for double Q-learning update step?

I've already read the original paper about double DQN but I do not find a clear and practical explanation of how the target $y$ is computed, so here's how I interpreted the method (let's say I have 3 ...
unter_983's user avatar
  • 331
11 votes
1 answer
4k views

What exactly is the advantage of double DQN over DQN?

I started looking into the double DQN (DDQN). Apparently, the difference between DDQN and DQN is that in DDQN we use the main value network for action selection and the target network for outputting ...
Chukwudi's user avatar
  • 359
2 votes
1 answer
481 views

How does the target network in double DQNs find the maximum Q value for each action?

I understand the fact that the neural network is used to take the states as inputs and it outputs the Q-value for state-action pairs. However, in order to compute this and update its weights, we need ...
Metrician's user avatar
1 vote
1 answer
251 views

Why does adding another network help in double DQN? [duplicate]

What is the idea behind double DQN? The target in double DQN is computed as follows $$ Y_{t}^{\text {DoubleQ }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1},...
joseph's user avatar
  • 111
4 votes
1 answer
125 views

What does the notation $p_t = \text{max}_{i<t} p_i$ mean in algorithm 1 of the prioritized experience replay paper?

I am having a hard time converting line 6 of the prioritized experience replay algorithm from the original paper into plain English (see below): I understand that new transitions (not visited before) ...
Hanzy's user avatar
  • 519
8 votes
2 answers
2k views

Can DQN perform better than Double DQN?

I'm training both DQN and double DQN in the same environment, but DQN performs significantly better than double DQN. As I've seen in the double DQN paper, double DQN should perform better than DQN. Am ...
Angelo's user avatar
  • 211
4 votes
1 answer
2k views

How can I ensure convergence of DDQN, if the true Q-values for different actions in the same state are very close?

I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future ...
apitsch's user avatar
  • 93