Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
22 votes
3 answers
5k views

Why doesn't Q-learning converge when using function approximation?

The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions (the Robbins-Monro conditions) regarding the learning rate are satisfied $\...
nbro's user avatar
  • 40.5k
17 votes
1 answer
6k views

Why does DQN require two different networks?

I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...
amitection's user avatar
25 votes
2 answers
12k views

Are there other approaches to deal with variable action spaces?

This question is about Reinforcement Learning and variable action spaces for every/some states. Variable action space Let's say you have an MDP, where the number of actions varies between states (for ...
Rikard Olsson's user avatar
18 votes
1 answer
19k views

How does LSTM in deep reinforcement learning differ from experience replay?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the author processed the Atari game frames with an LSTM layer at the end. My questions are: How does this method differ from the ...
Kevin. Fang's user avatar
8 votes
2 answers
961 views

What is experience replay in laymen's terms?

I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...
user491626's user avatar
2 votes
2 answers
550 views

How should I define the reward function to solve the Wumpus game with deep Q-learning?

I'm writing a DQN agent for the Wumpus game. Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...
Edwin Carlsson's user avatar
8 votes
2 answers
481 views

What are some online courses for deep reinforcement learning?

What are some (good) online courses for deep reinforcement learning? I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
J.Doe's user avatar
  • 91
10 votes
3 answers
3k views

How can you represent the state and action spaces for a card game in the case of a variable number of cards and actions?

I know how a machine can learn to play Atari games (Breakout): Playing Atari with Reinforcement Learning. With the same technique, it is even possible to play FPS games (Doom): Playing FPS Games with ...
Stefe Klauou's user avatar
8 votes
2 answers
1k views

Is reinforcement learning using shallow neural networks still deep reinforcement learning?

Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep. For example, PPO is often considered a ...
yewang's user avatar
  • 361
3 votes
3 answers
884 views

What is the optimal score for Tic Tac Toe for a reinforcement learning agent against a random opponent?

I guess this problem is encountered by everyone trying to solve Tic Tac Toe with various flavors of reinforcement learning. The answer is not "always win" because the random opponent may ...
Yan King Yin's user avatar
14 votes
2 answers
15k views

How large should the replay buffer be?

I'm learning DDPG algorithm by following the following link: Open AI Spinning Up document on DDPG, where it is written In order for the algorithm to have stable behavior, the replay buffer should ...
ycenycute's user avatar
  • 341
12 votes
1 answer
5k views

What exactly is the advantage of double DQN over DQN?

I started looking into the double DQN (DDQN). Apparently, the difference between DDQN and DQN is that in DDQN we use the main value network for action selection and the target network for outputting ...
Chukwudi's user avatar
  • 369
10 votes
2 answers
639 views

Was DeepMind's DQN learning simultaneously all the Atari games?

DeepMind states that its deep Q-network (DQN) was able to continually adapt its behavior while learning to play 49 Atari games. After learning all games with the same neural net, was the agent able to ...
Dion's user avatar
  • 203
8 votes
1 answer
6k views

Suitable reward function for trading buy and sell orders

I am working to build a deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are ...
fgauth's user avatar
  • 189
8 votes
2 answers
2k views

Can DQN perform better than Double DQN?

I'm training both DQN and double DQN in the same environment, but DQN performs significantly better than double DQN. As I've seen in the double DQN paper, double DQN should perform better than DQN. Am ...
Angelo's user avatar
  • 211
7 votes
1 answer
301 views

Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?

While reading the DQN paper, I found that randomly selecting and learning samples reduced divergence in RL using a non-linear function approximator (e.g a neural network). So, why does Reinforcement ...
강문주's user avatar
7 votes
1 answer
171 views

In imitation learning, do you simply inject optimal tuples of experience $(s, a, r, s')$ into your experience replay buffer?

Due to my RL algorithm having difficulties learning some control actions, I've decided to use imitation learning/apprenticeship learning to guide my RL to perform the optimal actions. I've read a few ...
Rui Nian's user avatar
  • 433
6 votes
2 answers
3k views

Why does self-playing tic-tac-toe not become perfect?

I trained a DQN that learns tic-tac-toe by playing against itself with a reward of -1/0/+1 for a loss/draw/win. Every 500 episodes, I test the progress by letting it play some episodes (also 500) ...
user3877351's user avatar
5 votes
0 answers
969 views

What could be causing the drastic performance drop of the DQN model on the Pong environment?

I am running a basic DQN (Deep Q-Network) on the Pong environment. Not a CNN, just a 3 layer linear neural net with ReLUs. It seems to work for the most part, but at some point, my model suffers from ...
Muppet's user avatar
  • 151
5 votes
1 answer
640 views

Clarifying representation of Neural Nerwork input for Chess Alpha Zero

In the Alpha Zero paper (https://arxiv.org/pdf/1712.01815.pdf) page 13, the input for the NN is described. In the beggining of the page, the authors state that: "The input to the Neural Network ...
Andrew's user avatar
  • 63
5 votes
2 answers
270 views

DQN arXiv 10-year anniversary: What are the outstanding problems being actively researched in deep Q-learning since 2019?

Background As of today (12-19-2023), the arXiv submission of the original deep Q-learning approach to achieve superhuman performance on ATARI games has turned a decade old. The original approach, ...
DeepQZero's user avatar
  • 1,379
5 votes
1 answer
4k views

Why do we need target network in deep Q learning? [duplicate]

I already know deep RL, but to learn it deeply I want to know why do we need 2 networks in deep RL. What does the target network do? I now there is huge mathematics into this, but I want to know deep ...
dato nefaridze's user avatar
5 votes
1 answer
1k views

How do we compute the target value when the agent ends up in the terminal state?

I am working on a deep reinforcement learning problem. Throughout the episode, there is a small positive and negative reward for good or bad decisions. In the end, there is a huge reward for the ...
pranav's user avatar
  • 191
4 votes
0 answers
2k views

Why is it hard to prove the convergence of the deep Q-learning algorithm?

Why is it hard to prove the convergence of the DQN algorithm? We know that the tabular Q-learning algorithm converges to the optimal Q-values, and with a linear approximator convergence is proved. ...
Afshin Oroojlooy's user avatar
3 votes
1 answer
1k views

Why does Deep Q Network outputs multiple Q values?

I am learning Deep RL following this tutorial: https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8 I understand everything but one detail: This image shows ...
NMO's user avatar
  • 133
3 votes
0 answers
254 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
Savco's user avatar
  • 61
2 votes
1 answer
1k views

How to train a reinforcement learning agent from raw pixels?

How would you train a reinforcement learning agent from raw pixels? For example, if you have 3 stacked images to sense motion, then how would you pass them to neural networks to output Q-learning ...
dato nefaridze's user avatar
2 votes
1 answer
740 views

How does sharing parameters between the policy and value functions help in PPO?

The PPO objective may include a value function error term when parameters are shared between the policy and value functions. How does this help, and when to use a neural network architecture that ...
Mika's user avatar
  • 341
2 votes
1 answer
207 views

How does one normalize observations in online reinforcement learning

I was wondering how would one normalize observations to a policy without knowing the upper and lower limits of the environment values. A trivial technique would be normalize each observation by its ...
desert_ranger's user avatar
2 votes
1 answer
236 views

Can I train an agent with DQN, avoiding obstacles and still finding the optimal path to finish the task?

The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible without ...
Mamora's user avatar
  • 53
2 votes
1 answer
131 views

Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents

Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for ...
Luca Anzalone's user avatar
2 votes
3 answers
431 views

Does AlphaGo play random moves in a real competition?

Alphago and AlphaGo zero use random play to generate data and use the data to train DNN. "Random play" means that there is a positive probability for AlphaGo to play some suboptimal moves ...
High GPA's user avatar
  • 163
2 votes
1 answer
253 views

Will the target network, which is less trained than the normal network, output inferior estimates?

I'm having some trouble understanding some parts of the usage of target networks. I get that having the same network predict the state/action/advantage values for both the current networks can lead ...
Alex's user avatar
  • 137
2 votes
2 answers
580 views

What is the target output for updating a Deep Q Network

I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space. The algorithm for table-based Q-Learning updates a single entry of the Q table - ...
Kricket's user avatar
  • 197
1 vote
1 answer
256 views

Why does adding another network help in double DQN? [duplicate]

What is the idea behind double DQN? The target in double DQN is computed as follows $$ Y_{t}^{\text {DoubleQ }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1},...
joseph's user avatar
  • 111
1 vote
0 answers
95 views

Can Q-learning be used for my scenario, and how might I do so?

I have already asked 2-3 general questions w.r.t Q learning and now I am asking a scenario specific one. I will try to be concise and understandable. I really really need help. Scenario: I have a ...
knowledge_seeker's user avatar
1 vote
1 answer
252 views

How to build a Neural Network to approximate the Q-function?

I am learning reinforcement learning with Q-learning using online resources, like blog posts, youtube videos, and books. At this point, I have learned the underpinning concepts of reinforcement ...
Exploring's user avatar
  • 343
0 votes
1 answer
255 views

How to compare memory requirements for tabular Q-learning vs deep neural network?

I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...
knowledge_seeker's user avatar