13
votes
Accepted
Why does DQN require two different networks?
My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one.
It has no real impact on computation ...
- 25.4k
12
votes
What is the difference between Q-learning, Deep Q-learning and Deep Q-network?
Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN).
Tabular Q-learning (TQL)
Deep Q-learning (DQL)
...
- 36.3k
10
votes
Can Q-learning be used for continuous (state or action) spaces?
Q-learning for continuous state spaces
Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the ...
- 25.4k
9
votes
Accepted
What exactly is the advantage of double DQN over DQN?
In $Q$-learning there is what is known as a maximisation bias. That is because the update target is $r + \gamma \max_a Q(s,a)$. If you slightly overestimate your $Q$-value then this error gets ...
- 4,340
8
votes
Accepted
Is Experience Replay like dreaming?
The speaker argued that a dream is a random addition of memories, just as experience replay.
The speaker is taking some liberties due to a general lack of scientific understanding of what dreams are. ...
- 25.4k
8
votes
Accepted
How to combine backpropagation in neural nets and reinforcement learning?
Gradient descent and back-propagation
In deep learning, gradient descent (GD) and back-propagation (BP) are used to update the weights of the neural network.
In reinforcement learning, one could map (...
8
votes
Accepted
Are policy gradient methods good for large discrete action spaces?
I don't think that (at least from a practical standpoint), there is much difference between continuous action space and discrete action space with >2k discrete actions. Quoting the "Continuous ...
- 1,972
7
votes
Accepted
My DQN is stuck and can't see where the problem is
After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were ...
- 173
7
votes
Accepted
What is the difference between Q-learning, Deep Q-learning and Deep Q-network?
In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, $Q(s, a)$. This is defined as
$$Q(s, a) = \mathbb{E}_\pi\left[ G_t | S_t = s, ...
- 4,340
6
votes
Accepted
How should I model all available actions of a chess game in deep Q-learning?
To model chess as a Markov decision problem (MDP) you can refer to the AlphaZero paper (Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm). The exact details can ...
- 522
6
votes
Accepted
What is the difference between DQN and AlphaGo Zero?
DQN and AlphaZero do not share much in terms of implementation.
However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, ...
- 25.4k
6
votes
Which kind of prioritized experience replay should I use?
The authors of that paper hypothesized that rank-based prioritization would be more robust to outliers. They suggested that rank-based sampling would be preferred for this reason. However, as they ...
- 2,008
6
votes
Accepted
What are some online courses for deep reinforcement learning?
Let me first say that deep RL is just the combination of RL with deep learning. So, if you study RL and deep learning, then studying deep RL should be straightforward. For this reason, this answer ...
- 36.3k
6
votes
Accepted
Can I apply DQN or policy gradient algorithms in the contextual bandit setting?
MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance.
Therefore, methods used in RL to solve ...
- 25.4k
6
votes
Why do we need target network in deep Q learning?
In DQN that was presented in the original paper the update target for the Q-Network is $\left(r_t + \max_aQ(s_{t+1},a;\theta^-) - Q(s_t,a_t; \theta)\right)^2$ were $\theta^-$ is some old version of ...
- 4,340
5
votes
Accepted
Is the DQN only applicable with images as inputs?
More precisely: is DQNN applicable only when we have high translational invariance in our input(s)?
No, DQN is not restricted to images or other kinds of inputs with those properties, it can be used ...
- 9,649
5
votes
Accepted
When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?
Gradient descent should be performed using the sum (or average) of the losses in the minibatch.
This is in fact also how I read the pseudocode in your question, though I understand it can be confusing....
- 9,649
5
votes
Accepted
What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?
1) Is there any way to set the initial Q-values for the actions?
You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network ...
- 896
5
votes
Why are reinforcement learning methods sample inefficient?
I will try to give a broad answer, if it's not helpful I'll remove it.
When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model ...
- 4,693
5
votes
What are some online courses for deep reinforcement learning?
For the programming part I suggest this YouTube channel by Phil Tabor (he also has a website: neuralnet.ai. I found his videos really useful while I was attending reinforcement learning classes at the ...
- 4,693
5
votes
Accepted
How and when should we update the Q-target in deep Q-learning?
The update form $\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$ (where $\theta'$ and $\theta$ represent the weights of the target network and the current network, respectively) does ...
- 722
5
votes
Accepted
How does one know that a problem is "model-free" in reinforcement learning?
Q-learning is said to be "model-free". Given the two examples above, is it because neither the lake's topology nor that of the mountain are changed by the actions taken?
No. That's not why ...
- 36.3k
5
votes
Accepted
Why do Q-values diverge without a target network?
I don't see how the target Q-value gets updated when the current Q-value is changed.
Without a separate target network, this happens because the approximator will generalise, and the generalisation ...
- 25.4k
4
votes
What are good learning strategies for Deep Q-Network with opponents?
In a two player zero-sum game (if I win, you lose and vice-versa), then you can have a simple and efficient solution learning from self-play.
How should an opponent be implemented to get good and ...
- 25.4k
4
votes
Why does self-playing tic-tac-toe not become perfect?
There are lots of ways that RL agents can fail to learn properly, so you are faced with a little bit of experimentation and maybe bug hunting unfortunately. However, from the description you have ...
- 25.4k
4
votes
Accepted
Can DQN announce it has things in its hand in a card game?
The simplest thing to do when you make you first implementation of the agent, is to automate decisions like this, in order to keep representations and decisions simple.
However, if you want to ...
- 25.4k
4
votes
Accepted
What is a high dimensional state in reinforcement learning?
Usually when people write about having a high-dimensional state space, they are referring to the state space actually used by the algorithm.
Suppose my state is a high dimensional vector of $N$ ...
- 9,649
4
votes
Accepted
Why does Deep Q Network outputs multiple Q values?
I think this was just a "clever" design choice. You can actually design a neural network (NN), to represent your Q function, which receives as input the state and an action and outputs the ...
- 36.3k
4
votes
Accepted
Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?
In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do ...
- 9,649
4
votes
Accepted
How do we compute the target value when the agent ends up in the terminal state?
Now, let us assume the agent is in the penultimate state, $S_1$, and
chooses the action $A$ that leads him to the completion state, $S_2$,
and gets a reward $R$.
How do we form the target value $Q_\...
- 25.4k
Only top scored, non community-wiki answers of a minimum length are eligible