# Tag Info

Accepted

### What exactly is the advantage of double DQN over DQN?

In $Q$-learning there is what is known as a maximisation bias. That is because the update target is $r + \gamma \max_a Q(s,a)$. If you slightly overestimate your $Q$-value then this error gets ...
• 4,920
Accepted

### Finding the true Q-values in gymnaiusm

In simple environments, the gold standard for true values to arbitrary accuracy would be to use dynamic programming, either policy iteration or value iteration (to evaluate a fixed policy, then use a ...
• 32.7k
Accepted

### How can I ensure convergence of DDQN, if the true Q-values for different actions in the same state are very close?

Let $Q^*(s, a)$ denote the "true" $Q$-value for a state-action pair $(s, a)$, i.e. the values that we're hoping to learn to approximate using a neural network that outputs $Q(s, a)$ values. The ...
• 10.3k
Accepted

### Does the DoubleDQN algorithm use a target network or two separate policies?

The canonical DoubleDQN uses the target network. I've not seen the first version used anywhere in the deep RL literature, but it looks like what one would do if they were to take the original Double Q-...
• 4,920

### Why does regular Q-learning (and DQN) overestimate the Q values?

The overestimation comes from the random initialisation of your Q-value estimates. Obviously these will not be perfect (if they were then we wouldn't need to learn the true Q-values!). In many value ...
• 4,920
Accepted

### What does the notation $p_t = \text{max}_{i<t} p_i$ mean in algorithm 1 of the prioritized experience replay paper?

From my interpretation what it means is that $p_t$ is the priority value associated with each transition and $p_t = max_{i<t} p_i$ means that the priority of transition number $t$ will be the ...
Accepted

### Can DQN perform better than Double DQN?

There is no thorough proof, theoretical or experimental that Double DQN is better then vanilla DQN. There are a lot of different tasks, paper and later experiments only explore some of them. What ...
• 725

### How Come My (D)DQN Fails To Learn?

I have two suggestions that you can look into. Based on my own work in RL, I believe the first one will require less work to implement. If the observability of the environment is not an issue, then ...
• 189
1 vote
Accepted

### Does "number of actions" refer to the number of actions taken or size of the action space?

The expression "number of actions" is being used in the same way in both cases. In fact, the letter $m$ is used in both cases. The number of actions (in the state $s$) is the number of ...
• 40.9k
1 vote

### DQN rgb input channels problem using pytorch

You can reshape it to (12, H, W) using NumPy reshape function. By the way, this will only increase the complexity of this problem. If you want to practice RL then just get the idea from their code and ...
1 vote

• 379
1 vote
Accepted

### How does the target network in double DQNs find the maximum Q value for each action?

Both in DQN and in DDQN, the target network starts as an exact copy of the Q-network, that has the same weights, layers, input and output dimensions, etc., as the Q-network. The main idea of the DQN ...
• 919
1 vote

### Why does adding another network help in double DQN?

As the authors of this paper state it: In $Q$-learning, the agent updates the value of executing an action in the current state, using the values of executing actions in a successive state. This ...
• 825
1 vote

### Can DQN perform better than Double DQN?

That may happen when the value of the state is bad. You can find the example and explain about that in the link below. See this:https://medium.freecodecamp.org/improvements-in-deep-q-learning-dueling-...

Only top scored, non community-wiki answers of a minimum length are eligible