Questions tagged [deep-rl]
For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.
38
questions
22
votes
3
answers
5k
views
Why doesn't Q-learning converge when using function approximation?
The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions (the Robbins-Monro conditions) regarding the learning rate are satisfied
$\...
17
votes
1
answer
6k
views
Why does DQN require two different networks?
I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...
25
votes
2
answers
12k
views
Are there other approaches to deal with variable action spaces?
This question is about Reinforcement Learning and variable action spaces for every/some states.
Variable action space
Let's say you have an MDP, where the number of actions varies between states (for ...
18
votes
1
answer
19k
views
How does LSTM in deep reinforcement learning differ from experience replay?
In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the author processed the Atari game frames with an LSTM layer at the end. My questions are:
How does this method differ from the ...
8
votes
2
answers
961
views
What is experience replay in laymen's terms?
I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...
2
votes
2
answers
550
views
How should I define the reward function to solve the Wumpus game with deep Q-learning?
I'm writing a DQN agent for the Wumpus game.
Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...
8
votes
2
answers
481
views
What are some online courses for deep reinforcement learning?
What are some (good) online courses for deep reinforcement learning?
I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
10
votes
3
answers
3k
views
How can you represent the state and action spaces for a card game in the case of a variable number of cards and actions?
I know how a machine can learn to play Atari games (Breakout): Playing Atari with Reinforcement Learning. With the same technique, it is even possible to play FPS games (Doom): Playing FPS Games with ...
8
votes
2
answers
1k
views
Is reinforcement learning using shallow neural networks still deep reinforcement learning?
Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep.
For example, PPO is often considered a ...
3
votes
3
answers
884
views
What is the optimal score for Tic Tac Toe for a reinforcement learning agent against a random opponent?
I guess this problem is encountered by everyone trying to solve Tic Tac Toe with various flavors of reinforcement learning.
The answer is not "always win" because the random opponent may ...
14
votes
2
answers
15k
views
How large should the replay buffer be?
I'm learning DDPG algorithm by following the following link: Open AI Spinning Up document on DDPG, where it is written
In order for the algorithm to have stable behavior, the replay buffer should ...
12
votes
1
answer
5k
views
What exactly is the advantage of double DQN over DQN?
I started looking into the double DQN (DDQN). Apparently, the difference between DDQN and DQN is that in DDQN we use the main value network for action selection and the target network for outputting ...
10
votes
2
answers
639
views
Was DeepMind's DQN learning simultaneously all the Atari games?
DeepMind states that its deep Q-network (DQN) was able to continually adapt its behavior while learning to play 49 Atari games.
After learning all games with the same neural net, was the agent able to ...
8
votes
1
answer
6k
views
Suitable reward function for trading buy and sell orders
I am working to build a deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are ...
8
votes
2
answers
2k
views
Can DQN perform better than Double DQN?
I'm training both DQN and double DQN in the same environment, but DQN performs significantly better than double DQN. As I've seen in the double DQN paper, double DQN should perform better than DQN. Am ...
7
votes
1
answer
301
views
Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?
While reading the DQN paper, I found that randomly selecting and learning samples reduced divergence in RL using a non-linear function approximator (e.g a neural network).
So, why does Reinforcement ...
7
votes
1
answer
171
views
In imitation learning, do you simply inject optimal tuples of experience $(s, a, r, s')$ into your experience replay buffer?
Due to my RL algorithm having difficulties learning some control actions, I've decided to use imitation learning/apprenticeship learning to guide my RL to perform the optimal actions. I've read a few ...
6
votes
2
answers
3k
views
Why does self-playing tic-tac-toe not become perfect?
I trained a DQN that learns tic-tac-toe by playing against itself with a reward of -1/0/+1 for a loss/draw/win. Every 500 episodes, I test the progress by letting it play some episodes (also 500) ...
5
votes
0
answers
969
views
What could be causing the drastic performance drop of the DQN model on the Pong environment?
I am running a basic DQN (Deep Q-Network) on the Pong environment. Not a CNN, just a 3 layer linear neural net with ReLUs.
It seems to work for the most part, but at some point, my model suffers from ...
5
votes
1
answer
640
views
Clarifying representation of Neural Nerwork input for Chess Alpha Zero
In the Alpha Zero paper (https://arxiv.org/pdf/1712.01815.pdf) page 13, the input for the NN is described. In the beggining of the page, the authors state that:
"The input to the Neural Network ...
5
votes
2
answers
270
views
DQN arXiv 10-year anniversary: What are the outstanding problems being actively researched in deep Q-learning since 2019?
Background
As of today (12-19-2023), the arXiv submission of the original deep Q-learning approach to achieve superhuman performance on ATARI games has turned a decade old. The original approach, ...
5
votes
1
answer
4k
views
Why do we need target network in deep Q learning? [duplicate]
I already know deep RL, but to learn it deeply I want to know why do we need 2 networks in deep RL. What does the target network do? I now there is huge mathematics into this, but I want to know deep ...
5
votes
1
answer
1k
views
How do we compute the target value when the agent ends up in the terminal state?
I am working on a deep reinforcement learning problem. Throughout the episode, there is a small positive and negative reward for good or bad decisions. In the end, there is a huge reward for the ...
4
votes
0
answers
2k
views
Why is it hard to prove the convergence of the deep Q-learning algorithm?
Why is it hard to prove the convergence of the DQN algorithm? We know that the tabular Q-learning algorithm converges to the optimal Q-values, and with a linear approximator convergence is proved.
...
3
votes
1
answer
1k
views
Why does Deep Q Network outputs multiple Q values?
I am learning Deep RL following this tutorial: https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8
I understand everything but one detail:
This image shows ...
3
votes
0
answers
254
views
What can be considered a deep recurrent neural network?
In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM.
I have DQN ...
2
votes
1
answer
1k
views
How to train a reinforcement learning agent from raw pixels?
How would you train a reinforcement learning agent from raw pixels?
For example, if you have 3 stacked images to sense motion, then how would you pass them to neural networks to output Q-learning ...
2
votes
1
answer
740
views
How does sharing parameters between the policy and value functions help in PPO?
The PPO objective may include a value function error term when parameters are shared between the policy and value functions. How does this help, and when to use a neural network architecture that ...
2
votes
1
answer
207
views
How does one normalize observations in online reinforcement learning
I was wondering how would one normalize observations to a policy without knowing the upper and lower limits of the environment values. A trivial technique would be normalize each observation by its ...
2
votes
1
answer
236
views
Can I train an agent with DQN, avoiding obstacles and still finding the optimal path to finish the task?
The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible without ...
2
votes
1
answer
131
views
Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents
Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for ...
2
votes
3
answers
431
views
Does AlphaGo play random moves in a real competition?
Alphago and AlphaGo zero use random play to generate data and use the data to train DNN. "Random play" means that there is a positive probability for AlphaGo to play some suboptimal moves ...
2
votes
1
answer
253
views
Will the target network, which is less trained than the normal network, output inferior estimates?
I'm having some trouble understanding some parts of the usage of target networks.
I get that having the same network predict the state/action/advantage values for both the current networks can lead ...
2
votes
2
answers
580
views
What is the target output for updating a Deep Q Network
I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space.
The algorithm for table-based Q-Learning updates a single entry of the Q table - ...
1
vote
1
answer
256
views
Why does adding another network help in double DQN? [duplicate]
What is the idea behind double DQN?
The target in double DQN is computed as follows
$$
Y_{t}^{\text {DoubleQ }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1},...
1
vote
0
answers
95
views
Can Q-learning be used for my scenario, and how might I do so?
I have already asked 2-3 general questions w.r.t Q learning and now I am asking a scenario specific one. I will try to be concise and understandable. I really really need help.
Scenario: I have a ...
1
vote
1
answer
252
views
How to build a Neural Network to approximate the Q-function?
I am learning reinforcement learning with Q-learning using online resources, like blog posts, youtube videos, and books. At this point, I have learned the underpinning concepts of reinforcement ...
0
votes
1
answer
255
views
How to compare memory requirements for tabular Q-learning vs deep neural network?
I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...