Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
13 votes
Accepted

Why does DQN require two different networks?

My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one. It has no real impact on computation ...
Neil Slater's user avatar
  • 32.4k
13 votes

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN). Tabular Q-learning (TQL) Deep Q-learning (DQL) ...
nbro's user avatar
  • 40.8k
12 votes

Can Q-learning be used for continuous (state or action) spaces?

Q-learning for continuous state spaces Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the ...
Neil Slater's user avatar
  • 32.4k
11 votes
Accepted

What exactly is the advantage of double DQN over DQN?

In $Q$-learning there is what is known as a maximisation bias. That is because the update target is $r + \gamma \max_a Q(s,a)$. If you slightly overestimate your $Q$-value then this error gets ...
David's user avatar
  • 4,910
8 votes
Accepted

Is Experience Replay like dreaming?

The speaker argued that a dream is a random addition of memories, just as experience replay. The speaker is taking some liberties due to a general lack of scientific understanding of what dreams are. ...
Neil Slater's user avatar
  • 32.4k
8 votes
Accepted

How to combine backpropagation in neural nets and reinforcement learning?

Gradient descent and back-propagation In deep learning, gradient descent (GD) and back-propagation (BP) are used to update the weights of the neural network. In reinforcement learning, one could map (...
8 votes
Accepted

How should I model all available actions of a chess game in deep Q-learning?

To model chess as a Markov decision problem (MDP) you can refer to the AlphaZero paper (Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm). The exact details can ...
Hai Nguyen's user avatar
8 votes
Accepted

Are policy gradient methods good for large discrete action spaces?

I don't think that (at least from a practical standpoint), there is much difference between continuous action space and discrete action space with >2k discrete actions. Quoting the "Continuous ...
Kostya's user avatar
  • 2,534
7 votes
Accepted

My DQN is stuck and can't see where the problem is

After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were ...
JCP's user avatar
  • 173
7 votes
Accepted

What are some online courses for deep reinforcement learning?

Let me first say that deep RL is just the combination of RL with deep learning. So, if you study RL and deep learning, then studying deep RL should be straightforward. For this reason, this answer ...
nbro's user avatar
  • 40.8k
7 votes

Why do we need target network in deep Q learning?

In DQN that was presented in the original paper the update target for the Q-Network is $\left(r_t + \max_aQ(s_{t+1},a;\theta^-) - Q(s_t,a_t; \theta)\right)^2$ were $\theta^-$ is some old version of ...
David's user avatar
  • 4,910
7 votes
Accepted

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, $Q(s, a)$. This is defined as $$Q(s, a) = \mathbb{E}_\pi\left[ G_t | S_t = s, ...
David's user avatar
  • 4,910
6 votes
Accepted

What is the difference between DQN and AlphaGo Zero?

DQN and AlphaZero do not share much in terms of implementation. However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, ...
Neil Slater's user avatar
  • 32.4k
6 votes

Which kind of prioritized experience replay should I use?

The authors of that paper hypothesized that rank-based prioritization would be more robust to outliers. They suggested that rank-based sampling would be preferred for this reason. However, as they ...
Philip Raeisghasem's user avatar
6 votes
Accepted

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance. Therefore, methods used in RL to solve ...
Neil Slater's user avatar
  • 32.4k
6 votes
Accepted

Why do my rewards reduce after extensive training using D3QN?

It is not 100% clear, but this seems like an instance of catastrophic forgetting. This is something that often impacts reinforcement learning. I have answered a very similar question on Data Science ...
Neil Slater's user avatar
  • 32.4k
5 votes
Accepted

Is the DQN only applicable with images as inputs?

More precisely: is DQNN applicable only when we have high translational invariance in our input(s)? No, DQN is not restricted to images or other kinds of inputs with those properties, it can be used ...
Dennis Soemers's user avatar
  • 10.3k
5 votes
Accepted

When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?

Gradient descent should be performed using the sum (or average) of the losses in the minibatch. This is in fact also how I read the pseudocode in your question, though I understand it can be confusing....
Dennis Soemers's user avatar
  • 10.3k
5 votes
Accepted

What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?

1) Is there any way to set the initial Q-values for the actions? You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network ...
BlueMoon93's user avatar
5 votes

Why are reinforcement learning methods sample inefficient?

I will try to give a broad answer, if it's not helpful I'll remove it. When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model ...
Edoardo Guerriero's user avatar
5 votes

What are some online courses for deep reinforcement learning?

For the programming part I suggest this YouTube channel by Phil Tabor (he also has a website: neuralnet.ai. I found his videos really useful while I was attending reinforcement learning classes at the ...
Edoardo Guerriero's user avatar
5 votes
Accepted

How and when should we update the Q-target in deep Q-learning?

The update form $\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$ (where $\theta'$ and $\theta$ represent the weights of the target network and the current network, respectively) does ...
user5093249's user avatar
5 votes
Accepted

Why do DQNs tend to forget?

You are referring to catastrophic forgetting which could be an issue in any neural net. More specifically for DQN refer to this article.
pedrum's user avatar
  • 313
5 votes
Accepted

How does one know that a problem is "model-free" in reinforcement learning?

Q-learning is said to be "model-free". Given the two examples above, is it because neither the lake's topology nor that of the mountain are changed by the actions taken? No. That's not why ...
nbro's user avatar
  • 40.8k
5 votes
Accepted

How to deal with changing environment in reinforcement learning

I am correct in my understanding that you only provide the agent with the state of the car, i.e. a global x and y position, its angle, velocity, and steering angle? How does the agent know that it is ...
Lars's user avatar
  • 189
5 votes
Accepted

Why do Q-values diverge without a target network?

I don't see how the target Q-value gets updated when the current Q-value is changed. Without a separate target network, this happens because the approximator will generalise, and the generalisation ...
Neil Slater's user avatar
  • 32.4k
5 votes
Accepted

Finding the true Q-values in gymnaiusm

In simple environments, the gold standard for true values to arbitrary accuracy would be to use dynamic programming, either policy iteration or value iteration (to evaluate a fixed policy, then use a ...
Neil Slater's user avatar
  • 32.4k
4 votes

What are good learning strategies for Deep Q-Network with opponents?

In a two player zero-sum game (if I win, you lose and vice-versa), then you can have a simple and efficient solution learning from self-play. How should an opponent be implemented to get good and ...
Neil Slater's user avatar
  • 32.4k
4 votes

Why does self-playing tic-tac-toe not become perfect?

There are lots of ways that RL agents can fail to learn properly, so you are faced with a little bit of experimentation and maybe bug hunting unfortunately. However, from the description you have ...
Neil Slater's user avatar
  • 32.4k
4 votes
Accepted

Can DQN announce it has things in its hand in a card game?

The simplest thing to do when you make you first implementation of the agent, is to automate decisions like this, in order to keep representations and decisions simple. However, if you want to ...
Neil Slater's user avatar
  • 32.4k

Only top scored, non community-wiki answers of a minimum length are eligible