Questions tagged [deep-rl]
For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.
20
questions
15
votes
3answers
2k views
Why doesn't Q-learning converge when using function approximation?
The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions (the Robbins-Monro conditions) regarding the learning rate are satisfied
$\...
14
votes
1answer
12k views
How does LSTM in deep reinforcement learning differ from experience replay?
In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the author processed the Atari game frames with an LSTM layer at the end. My questions are:
How does this method differ from the ...
14
votes
1answer
2k views
Why does DQN require two different networks?
I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...
5
votes
2answers
280 views
What are some online courses for deep reinforcement learning?
What are some (good) online courses for deep reinforcement learning?
I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
2
votes
1answer
237 views
How do we compute the target value when the agent ends up in the terminal state?
I am working on a deep reinforcement learning problem. Throughout the episode, there is a small positive and negative reward for good or bad decisions. In the end, there is a huge reward for the ...
7
votes
2answers
471 views
Is reinforcement learning using shallow neural networks still deep reinforcement learning?
Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep.
For example, PPO is often considered a ...
9
votes
2answers
477 views
What is experience replay in laymen's terms?
I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...
8
votes
3answers
1k views
How can you represent the state and action spaces for a card game in the case of a variable number of cards and actions?
I know how a machine can learn to play Atari games (Breakout): Playing Atari with Reinforcement Learning. With the same technique, it is even possible to play FPS games (Doom): Playing FPS Games with ...
2
votes
1answer
315 views
How to train a reinforcement learning agent from raw pixels?
How would you train a reinforcement learning agent from raw pixels?
For example, if you have 3 stacked images to sense motion, then how would you pass them to neural networks to output Q-learning ...
7
votes
1answer
122 views
Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?
While reading the DQN paper, I found that randomly selecting and learning samples reduced divergence in RL using a non-linear function approximator (e.g a neural network).
So, why does Reinforcement ...
6
votes
2answers
387 views
Can DQN perform better than Double DQN?
I'm training both DQN and double DQN in the same environment, but DQN performs significantly better than double DQN. As I've seen in the double DQN paper, double DQN should perform better than DQN. Am ...
3
votes
0answers
170 views
What can be considered a deep recurrent neural network?
In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM.
I have DQN ...
3
votes
1answer
442 views
Why does Deep Q Network outputs multiple Q values?
I am learning Deep RL following this tutorial: https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8
I understand everything but one detail:
This image shows ...
2
votes
1answer
190 views
What exactly is the advantage of double DQN over DQN?
I started looking into the double DQN (DDQN). Apparently, the difference between DDQN and DQN is that in DDQN we use the main value network for action selection and the target network for outputting ...
2
votes
2answers
122 views
How should I define the reward function to solve the Wumpus game with deep Q-learning?
I'm writing a DQN agent for the Wumpus game.
Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...
1
vote
0answers
69 views
What happens if our target network overestimates the value?
When we use DDQN, we often use the target network in case our online network overestimates a value, but this doesn't make sense to me, because
What happens if our target network is the one that ...
1
vote
1answer
49 views
How to build a Neural Network to approximate the Q-function?
I am learning reinforcement learning with Q-learning using online resources, like blog posts, youtube videos, and books. At this point, I have learned the underpinning concepts of reinforcement ...
1
vote
0answers
80 views
Why is it hard to prove the convergence of the deep Q-learning algorithm?
Why is it hard to prove the convergence of the DQN algorithm? We know that the tabular Q-learning algorithm converges to the optimal Q-values, and with a linear approximator convergence is proved.
...
1
vote
1answer
69 views
Why does adding another network help in double DQN? [duplicate]
What is the idea behind double DQN?
The target in double DQN is computed as follows
$$
Y_{t}^{\text {DoubleQ }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1},...
1
vote
2answers
92 views
What is the target output for updating a Deep Q Network
I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space.
The algorithm for table-based Q-Learning updates a single entry of the Q table - ...
