Hot answers tagged

13 votes
Accepted

Why does DQN require two different networks?

My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one. It has no real impact on computation ...
  • 25.4k
12 votes

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN). Tabular Q-learning (TQL) Deep Q-learning (DQL) ...
  • 36.3k
10 votes

Can Q-learning be used for continuous (state or action) spaces?

Q-learning for continuous state spaces Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the ...
  • 25.4k
9 votes
Accepted

What exactly is the advantage of double DQN over DQN?

In $Q$-learning there is what is known as a maximisation bias. That is because the update target is $r + \gamma \max_a Q(s,a)$. If you slightly overestimate your $Q$-value then this error gets ...
8 votes
Accepted

Is Experience Replay like dreaming?

The speaker argued that a dream is a random addition of memories, just as experience replay. The speaker is taking some liberties due to a general lack of scientific understanding of what dreams are. ...
  • 25.4k
8 votes
Accepted

How to combine backpropagation in neural nets and reinforcement learning?

Gradient descent and back-propagation In deep learning, gradient descent (GD) and back-propagation (BP) are used to update the weights of the neural network. In reinforcement learning, one could map (...
8 votes
Accepted

Are policy gradient methods good for large discrete action spaces?

I don't think that (at least from a practical standpoint), there is much difference between continuous action space and discrete action space with >2k discrete actions. Quoting the "Continuous ...
  • 1,972
7 votes
Accepted

My DQN is stuck and can't see where the problem is

After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were ...
  • 173
7 votes
Accepted

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, $Q(s, a)$. This is defined as $$Q(s, a) = \mathbb{E}_\pi\left[ G_t | S_t = s, ...
6 votes
Accepted

How should I model all available actions of a chess game in deep Q-learning?

To model chess as a Markov decision problem (MDP) you can refer to the AlphaZero paper (Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm). The exact details can ...
6 votes
Accepted

What is the difference between DQN and AlphaGo Zero?

DQN and AlphaZero do not share much in terms of implementation. However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, ...
  • 25.4k
6 votes

Which kind of prioritized experience replay should I use?

The authors of that paper hypothesized that rank-based prioritization would be more robust to outliers. They suggested that rank-based sampling would be preferred for this reason. However, as they ...
6 votes
Accepted

What are some online courses for deep reinforcement learning?

Let me first say that deep RL is just the combination of RL with deep learning. So, if you study RL and deep learning, then studying deep RL should be straightforward. For this reason, this answer ...
  • 36.3k
6 votes
Accepted

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance. Therefore, methods used in RL to solve ...
  • 25.4k
6 votes

Why do we need target network in deep Q learning?

In DQN that was presented in the original paper the update target for the Q-Network is $\left(r_t + \max_aQ(s_{t+1},a;\theta^-) - Q(s_t,a_t; \theta)\right)^2$ were $\theta^-$ is some old version of ...
5 votes
Accepted

Is the DQN only applicable with images as inputs?

More precisely: is DQNN applicable only when we have high translational invariance in our input(s)? No, DQN is not restricted to images or other kinds of inputs with those properties, it can be used ...
  • 9,649
5 votes
Accepted

When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?

Gradient descent should be performed using the sum (or average) of the losses in the minibatch. This is in fact also how I read the pseudocode in your question, though I understand it can be confusing....
  • 9,649
5 votes
Accepted

What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?

1) Is there any way to set the initial Q-values for the actions? You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network ...
5 votes

Why are reinforcement learning methods sample inefficient?

I will try to give a broad answer, if it's not helpful I'll remove it. When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model ...
5 votes

What are some online courses for deep reinforcement learning?

For the programming part I suggest this YouTube channel by Phil Tabor (he also has a website: neuralnet.ai. I found his videos really useful while I was attending reinforcement learning classes at the ...
5 votes
Accepted

How and when should we update the Q-target in deep Q-learning?

The update form $\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$ (where $\theta'$ and $\theta$ represent the weights of the target network and the current network, respectively) does ...
5 votes
Accepted

How does one know that a problem is "model-free" in reinforcement learning?

Q-learning is said to be "model-free". Given the two examples above, is it because neither the lake's topology nor that of the mountain are changed by the actions taken? No. That's not why ...
  • 36.3k
5 votes
Accepted

Why do Q-values diverge without a target network?

I don't see how the target Q-value gets updated when the current Q-value is changed. Without a separate target network, this happens because the approximator will generalise, and the generalisation ...
  • 25.4k
4 votes

What are good learning strategies for Deep Q-Network with opponents?

In a two player zero-sum game (if I win, you lose and vice-versa), then you can have a simple and efficient solution learning from self-play. How should an opponent be implemented to get good and ...
  • 25.4k
4 votes

Why does self-playing tic-tac-toe not become perfect?

There are lots of ways that RL agents can fail to learn properly, so you are faced with a little bit of experimentation and maybe bug hunting unfortunately. However, from the description you have ...
  • 25.4k
4 votes
Accepted

Can DQN announce it has things in its hand in a card game?

The simplest thing to do when you make you first implementation of the agent, is to automate decisions like this, in order to keep representations and decisions simple. However, if you want to ...
  • 25.4k
4 votes
Accepted

What is a high dimensional state in reinforcement learning?

Usually when people write about having a high-dimensional state space, they are referring to the state space actually used by the algorithm. Suppose my state is a high dimensional vector of $N$ ...
  • 9,649
4 votes
Accepted

Why does Deep Q Network outputs multiple Q values?

I think this was just a "clever" design choice. You can actually design a neural network (NN), to represent your Q function, which receives as input the state and an action and outputs the ...
  • 36.3k
4 votes
Accepted

Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?

In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do ...
  • 9,649
4 votes
Accepted

How do we compute the target value when the agent ends up in the terminal state?

Now, let us assume the agent is in the penultimate state, $S_1$, and chooses the action $A$ that leads him to the completion state, $S_2$, and gets a reward $R$. How do we form the target value $Q_\...
  • 25.4k

Only top scored, non community-wiki answers of a minimum length are eligible