# Tag Info

Accepted

### How does LSTM in deep reinforcement learning differ from experience replay?

How does this method differ from the experience replay, as they both use past information in the training? What's the typical application of both techniques? Using a recurrent neural network is one ...
• 24k

### Are there other approaches to deal with variable action spaces?

Does anyone know any paper regarding this subject? I'm not familiar with any off the top of my head. I do know that the vast majority of Reinforcement Learning literature focuses on settings with a ...
• 9,379

• 436
Accepted

### Do we have to use CNN for Deep Q Learning?

No. DQN and other deep RL methods work well with fully connected layers. Here's an implementation of DQN which doesn't use CNNs: github.com/keon/deep-q-learning/blob/master/dqn.py DeepMind mostly use ...
• 655

### How large should the replay buffer be?

In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. The larger the ...
• 33.8k

### What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN). Tabular Q-learning (TQL) Deep Q-learning (DQL) ...
• 33.8k
Accepted

### What are the major differences between multi-armed bandits and the other well-known algorithms (DQN, A3C, PPO, etc)?

You should start with the general definition of Reinforcement Learning problem. And what Markov Decision Process is. DQN, A3C, PPO and REINFORCE are algorithms for solving reinforcement learning ...
• 1,813
Accepted

### Is Experience Replay like dreaming?

The speaker argued that a dream is a random addition of memories, just as experience replay. The speaker is taking some liberties due to a general lack of scientific understanding of what dreams are. ...
• 24k
Accepted

### How to combine backpropagation in neural nets and reinforcement learning?

Gradient descent and back-propagation In deep learning, gradient descent (GD) and back-propagation (BP) are used to update the weights of the neural network. In reinforcement learning, one could map (...
Accepted

• 9,379
Accepted

### My DQN is stuck and can't see where the problem is

After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were ...
• 153
Accepted

### What is the difference between DQN and AlphaGo Zero?

DQN and AlphaZero do not share much in terms of implementation. However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, ...
• 24k

### Is there an alternative to the use of target network?

I have done some research and would like to share. Generally to eliminate the use of target network one needs to show that training would be stable under off-policy semi-gradient. There are two ...
• 490
Accepted

### What are some online courses for deep reinforcement learning?

Let me first say that deep RL is just the combination of RL with deep learning. So, if you study RL and deep learning, then studying deep RL should be straightforward. For this reason, this answer ...
• 33.8k
Accepted

### Why AlphaGo didn't use Deep Q-Learning?

$Q$-learning (and also its deep variant, and most of the other well-known reinforcement learning algorithms) are inherently learning approaches for single-agent environments. The entire problem ...
• 9,379
Accepted

### Clarifying representation of Neural Nerwork input for Chess Alpha Zero

For anyone wondering, I believe to have found the answer: Yes, it will be an 8x8 plane where all the entries are the same, the number of moves (or mpves with no progress). There are two repetitions ...
Accepted

### Is the DQN only applicable with images as inputs?

More precisely: is DQNN applicable only when we have high translational invariance in our input(s)? No, DQN is not restricted to images or other kinds of inputs with those properties, it can be used ...
• 9,379
Accepted

### When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?

Gradient descent should be performed using the sum (or average) of the losses in the minibatch. This is in fact also how I read the pseudocode in your question, though I understand it can be confusing....
• 9,379
We can start with equation (30): $$\bar{A}(s) = P(a \neq \tilde{a}) \mathbb{E}_{(a,\tilde{a})\sim(\pi,\tilde{\pi}|a\neq\tilde{a})} [A_\pi(s, \tilde{a}) - A_\pi(s, a)]$$ Taking the absolute value ...