12
votes
Accepted
Why does DQN require two different networks?
My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one.
It has no real impact on computation ...
- 23.3k
9
votes
What is the difference between Q-learning, Deep Q-learning and Deep Q-network?
Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN).
Tabular Q-learning (TQL)
Deep Q-learning (DQL)
...
- 33.2k
8
votes
Accepted
Is Experience Replay like dreaming?
The speaker argued that a dream is a random addition of memories, just as experience replay.
The speaker is taking some liberties due to a general lack of scientific understanding of what dreams are. ...
- 23.3k
8
votes
Accepted
How to combine backpropagation in neural nets and reinforcement learning?
Gradient descent and back-propagation
In deep learning, gradient descent (GD) and back-propagation (BP) are used to update the weights of the neural network.
In reinforcement learning, one could map (...
8
votes
Can Q-learning be used for continuous (state or action) spaces?
Q-learning for continuous state spaces
Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the ...
- 23.3k
6
votes
Accepted
How should I model all available actions of a chess game in deep Q-learning?
To model chess as a Markov decision problem (MDP) you can refer to the AlphaZero paper (Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm). The exact details can ...
- 522
6
votes
Accepted
My DQN is stuck and can't see where the problem is
After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were ...
- 153
6
votes
Accepted
What is the difference between DQN and AlphaGo Zero?
DQN and AlphaZero do not share much in terms of implementation.
However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, ...
- 23.3k
6
votes
Which kind of prioritized experience replay should I use?
The authors of that paper hypothesized that rank-based prioritization would be more robust to outliers. They suggested that rank-based sampling would be preferred for this reason. However, as they ...
- 1,928
6
votes
Accepted
What are some online courses for deep reinforcement learning?
Let me first say that deep RL is just the combination of RL with deep learning. So, if you study RL and deep learning, then studying deep RL should be straightforward. For this reason, this answer ...
- 33.2k
5
votes
Accepted
Is the DQN only applicable with images as inputs?
More precisely: is DQNN applicable only when we have high translational invariance in our input(s)?
No, DQN is not restricted to images or other kinds of inputs with those properties, it can be used ...
- 9,326
5
votes
Accepted
When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?
Gradient descent should be performed using the sum (or average) of the losses in the minibatch.
This is in fact also how I read the pseudocode in your question, though I understand it can be confusing....
- 9,326
5
votes
Accepted
What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?
1) Is there any way to set the initial Q-values for the actions?
You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network ...
- 856
5
votes
Why are reinforcement learning methods sample inefficient?
I will try to give a broad answer, if it's not helpful I'll remove it.
When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model ...
- 3,453
5
votes
What are some online courses for deep reinforcement learning?
For the programming part I suggest this YouTube channel by Phil Tabor (he also has a website: neuralnet.ai. I found his videos really useful while I was attending reinforcement learning classes at the ...
- 3,453
5
votes
Accepted
Can I apply DQN or policy gradient algorithms in the contextual bandit setting?
MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance.
Therefore, methods used in RL to solve ...
- 23.3k
5
votes
Accepted
What is the difference between Q-learning, Deep Q-learning and Deep Q-network?
In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, $Q(s, a)$. This is defined as
$$Q(s, a) = \mathbb{E}_\pi\left[ G_t | S_t = s, ...
- 4,036
4
votes
What are good learning strategies for Deep Q-Network with opponents?
In a two player zero-sum game (if I win, you lose and vice-versa), then you can have a simple and efficient solution learning from self-play.
How should an opponent be implemented to get good and ...
- 23.3k
4
votes
Why does self-playing tic-tac-toe not become perfect?
There are lots of ways that RL agents can fail to learn properly, so you are faced with a little bit of experimentation and maybe bug hunting unfortunately. However, from the description you have ...
- 23.3k
4
votes
Accepted
Can DQN announce it has things in its hand in a card game?
The simplest thing to do when you make you first implementation of the agent, is to automate decisions like this, in order to keep representations and decisions simple.
However, if you want to ...
- 23.3k
4
votes
Accepted
What is a high dimensional state in reinforcement learning?
Usually when people write about having a high-dimensional state space, they are referring to the state space actually used by the algorithm.
Suppose my state is a high dimensional vector of $N$ ...
- 9,326
4
votes
Accepted
Why does Deep Q Network outputs multiple Q values?
I think this was just a "clever" design choice. You can actually design a neural network (NN), to represent your Q function, which receives as input the state and an action and outputs the ...
- 33.2k
4
votes
Accepted
Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?
In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do ...
- 9,326
4
votes
Accepted
How do we compute the target value when the agent ends up in the terminal state?
Now, let us assume the agent is in the penultimate state, $S_1$, and
chooses the action $A$ that leads him to the completion state, $S_2$,
and gets a reward $R$.
How do we form the target value $Q_\...
- 23.3k
4
votes
Accepted
Should I grey-scale the coloured frames/channels to build the approximation of the state?
We can't say for sure which approach would work best in the general case. If you have domain knowledge, you can make a better guess. You'll basically want to answer the question: which information is ...
- 9,326
4
votes
Why are reinforcement learning methods sample inefficient?
This is mostly because humans already have information when they start learning the game (priors) that makes them learn it more quickly. We already know to jump on monsters or avoid them or to get ...
- 514
4
votes
Are Q values estimated from a DQN different from a duelling DQN with the same number of layers and filters?
Dueling-DQN has different network architecture comparing to vanilla DQN, so I don't think your version will work as well as the Dueling architecture.
From Wang et al., 2016, Dueling Network ...
- 226
4
votes
What is the target Q-value in DQNs?
When training a Deep Q network with experienced replay, you accumulate what is known as training experiences $e_t = (s_t, a_t, r_t, s_{t+1})$. You then sample a batch of such experiences and for each ...
- 1,201
4
votes
Accepted
How and when should we update the Q-target in deep Q-learning?
The update form $\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$ (where $\theta'$ and $\theta$ represent the weights of the target network and the current network, respectively) does ...
- 682
4
votes
Accepted
What happens when you select actions using softmax instead of epsilon greedy in DQN?
DQN on the other hand, explores using epsilon greedy exploration. Either selecting the best action or a random action.
This is a very common choice, because it is simple to implement and quite robust....
- 23.3k
Only top scored, non community-wiki answers of a minimum length are eligible