5 votes
Accepted

How can I ensure convergence of DDQN, if the true Q-values for different actions in the same state are very close?

Let $Q^*(s, a)$ denote the "true" $Q$-value for a state-action pair $(s, a)$, i.e. the values that we're hoping to learn to approximate using a neural network that outputs $Q(s, a)$ values. The ...
  • 9,519
5 votes
Accepted

What exactly is the advantage of double DQN over DQN?

In $Q$-learning there is what is known as a maximisation bias. That is because the update target is $r + \gamma \max_a Q(s,a)$. If you slightly overestimate your $Q$-value then this error gets ...
4 votes

Why does regular Q-learning (and DQN) overestimate the Q values?

The overestimation comes from the random initialisation of your Q-value estimates. Obviously these will not be perfect (if they were then we wouldn't need to learn the true Q-values!). In many value ...
3 votes
Accepted

What does the notation $p_t = \text{max}_{i<t} p_i$ mean in algorithm 1 of the prioritized experience replay paper?

From my interpretation what it means is that $p_t$ is the priority value associated with each transition and $p_t = max_{i<t} p_i $ means that the priority of transition number $t$ will be the ...
2 votes
Accepted

Can DQN perform better than Double DQN?

There is no thorough proof, theoretical or experimental that Double DQN is better then vanilla DQN. There are a lot of different tasks, paper and later experiments only explore some of them. What ...
2 votes

How Come My (D)DQN Fails To Learn?

I have two suggestions that you can look into. Based on my own work in RL, I believe the first one will require less work to implement. If the observability of the environment is not an issue, then ...
  • 179
1 vote
Accepted

Does "number of actions" refer to the number of actions taken or size of the action space?

The expression "number of actions" is being used in the same way in both cases. In fact, the letter $m$ is used in both cases. The number of actions (in the state $s$) is the number of ...
  • 34.9k
1 vote

DQN rgb input channels problem using pytorch

You can reshape it to (12, H, W) using NumPy reshape function. By the way, this will only increase the complexity of this problem. If you want to practice RL then just get the idea from their code and ...
1 vote

Why do we minimise the loss between the target Q values and 'local' Q values?

The loss function is designed in a way to approximate the bellman optimality for $Q^*(s,a)$. Given an optimal policy $\pi^*$, $Q^*(s,a)$ satisfies the equation $$Q^*(s,a) = r(s) + \gamma max_{a'}\sum_{...
  • 1,241
1 vote

How to compute the target for double Q-learning update step?

$$Y_{t}^{\text {DoubleDQN }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}\right), \boldsymbol{\theta}_{t}^{-}\right)$$ The only ...
  • 349
1 vote
Accepted

How does the target network in double DQNs find the maximum Q value for each action?

Both in DQN and in DDQN, the target network starts as an exact copy of the Q-network, that has the same weights, layers, input and output dimensions, etc., as the Q-network. The main idea of the DQN ...
  • 889
1 vote

Why does adding another network help in double DQN?

As the authors of this paper state it: In $Q$-learning, the agent updates the value of executing an action in the current state, using the values of executing actions in a successive state. This ...
  • 715
1 vote

Can DQN perform better than Double DQN?

That may happen when the value of the state is bad. You can find the example and explain about that in the link below. See this:https://medium.freecodecamp.org/improvements-in-deep-q-learning-dueling-...

Only top scored, non community-wiki answers of a minimum length are eligible