Questions tagged [dqn]

For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.

Filter by
Sorted by
Tagged with
14
votes
1answer
2k views

Why does DQN require two different networks?

I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...
8
votes
3answers
2k views

Huge action space size in Reinforcement Learning

I am working on creating a RL based AI for a certain board game. Just as a general overview of the game so that you understand what it's all about: It's a discrete turn-based game with a board of size ...
7
votes
1answer
103 views

Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?

While reading the DQN paper, I found that randomly selecting and learning samples reduced divergence in RL using a non-linear function approximator (e.g a neural network). So why does Reinforcement ...
6
votes
2answers
137 views

Why are reinforcement learning methods sample inefficient?

Reinforcement learning methods are considered to be extremely sample inefficient. For example, in a recent DeepMind paper by Hessel et al., they showed that in order to reach human-level performance ...
5
votes
2answers
195 views

What are some online courses for deep reinforcement learning?

What are some (good) online courses for deep reinforcement learning? I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
5
votes
2answers
944 views

What is the difference between DQN and AlphaGo Zero?

I have already implemented a relatively simple DQN on Pacman. Now I would like to clearly understand the difference between a DQN and the techniques used by AlphaGo zero/AlphaZero and I couldn't find ...
5
votes
1answer
366 views

Which kind of prioritized experience replay should I use?

The Prioritized Experience Replay paper gives two different ways of sampling from the replay buffer. One, called "proportional prioritization", assigns each transition a priority proportional to its ...
5
votes
1answer
82 views

Can exogenous variables be state features in reinforcement learning?

I have a question about state representation of Q-learning or DQN algorithm. I'm still a beginner of RL, so I'm not sure that is it suitable to take exogenous variables as state features. For example,...
5
votes
2answers
308 views

Why don't people use projected Bellman error with deep neural networks?

Projected Bellman error has shown to be stable with linear function approximation. The technique is not at all new. I can only wonder why this technique is not adopted to use with non-linear function ...
5
votes
2answers
256 views

Can DQN perform better than Double DQN?

I'm training both kind of agents against an environment but DQN performs significantly better than Double DQN. As I've saw here, Double DQN use to perform better than DQN. Am I doing something wrong ...
4
votes
3answers
410 views

What is the target Q-value in DQNs?

I understand that in DQNs, the loss is measured by taking the MSE of outputted Q-values and target Q-values. Whats does the target Q-values represent? And how is it obtained/calculated by the DQN?
4
votes
3answers
267 views

Upper limit to the maximum cumulative reward in a deep reinforcement learning problem

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem? For example you want to train an DQN agent in an environment and you want to know what is the highest ...
4
votes
1answer
902 views

Ensure convergence of DDQN if true Q-values are very close

I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future reward from ...
4
votes
1answer
180 views

Why do DQNs tend to forget?

Why do DQNs tend to forget? Is it because when you feed highly correlated samples, your model (function approximation) doesn't give a general solution? For example: I use level 1 experiences, my ...
4
votes
2answers
113 views

My Deep Q-Learning Network does not learn for OpenAI gym's cartpole problem

I am implementing OpenAI gym's cartpole problem using Deep Q-Learning (DQN). I followed tutorials (video and otherwise) and learned all about it. I implemented a code for myself and I thought it ...
4
votes
1answer
121 views

What happens when you select actions using softmax instead of epsilon greedy in DQN?

I understand the two major branches of RL are Q-Learning and Policy Gradient methods. From my understanding (correct me if I'm wrong), policy gradient methods have an inherent exploration built-in as ...
4
votes
1answer
286 views

Why does the DQN not converge when the start or goal states can change dynamically?

I'm trying to apply a DQN to a stochastic environment, but I'm having trouble getting it to converge. I found some similar questions asked here, but no solutions yet. I can easily get the DQN to ...
4
votes
1answer
103 views

How can a DQN backpropagate its loss?

I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I ...
4
votes
1answer
511 views

How is the gradient of the loss function in DQN derived?

In the original DQN paper, page 1, the loss function of the DQN is $$ L_{i}(\theta_{i}) = \mathbb{E}_{(s,a,r,s') \sim U(D)} [(r+\gamma \max_{a'} Q(s',a',\theta_{i}^{-}) - Q(s,a;\theta_{i}))^2] $$ ...
4
votes
1answer
54 views

How to represent players in a multi agent environment so each model can distinguish its own player

So I have 2 models trained with the DQN algorithm that I want to train in a multi-agent environment to see how they react with each other. The models were trained in an environment consisting of 0's ...
4
votes
0answers
319 views

What could be causing the drastic performance drop of the DQN model on the Pong environment?

I am running a basic DQN (Deep Q-Network) on the Pong environment. Not a CNN, just a 3 layer linear neural net with ReLUs. It seems to work for the most part, but at some point, my model suffers from ...
3
votes
2answers
961 views

Each training run for DDQN agent takes 2 days, and still ends up with -13 avg score, but OpenAi baseline DQN needs only an hour to converge to +18?

Status: For a few weeks now, I have been working on a Double DQN agent for the PongDeterministic-v4 environment, which you can find here. A single training run ...
3
votes
1answer
1k views

How should I model all available actions of a chess game in deep Q-learning?

I just read about deep Q-learning, which is using a neural network for the value function instead of a table. I saw the example here: https://yanpanlau.github.io/2016/07/10/FlappyBird-Keras.html and ...
3
votes
1answer
78 views

Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?

In the original prioritized experience replay paper, the authors track $\gamma_t$ in every state transition tuple (see line 6 in algorithm below): Why do the authors track this at every time step? ...
3
votes
2answers
544 views

What does it mean by high dimensional state in DQN?

Going through the DQN paper, it said the state-space is high dimensional. I am a little bit confused here. Suppose my state is a high dimensional vector of N length ...
3
votes
1answer
58 views

How would researchers determine the best deep learning model if every run of the code yields different results?

There are many factors that cause the results of ML models to be different for every run of the same piece of code. One factor could be different initialization of weights in the neural network. ...
3
votes
1answer
79 views

Are Q values estimated from a DQN different from a duelling DQN with the same number of layers and filters?

I am confused about the Q values of a duelling deep Q network (DQN). As far as I know, duelling DQNs have 2 outputs Advantage: how good it is to be in a particular state $s$ Value: the advantage of ...
3
votes
2answers
941 views

My DQN is stuck and can't see where the problem is

I'm trying to replicate the DeepMind paper results, so I implemented my own DQN. I left it training for more than 4 million frames (more than 2000 episodes) on SpaceInvaders-v4 (OpenAI-Gym) and it ...
3
votes
1answer
1k views

DQN input representation for a card game

In order to learn about DP and RL, I chose to start a side project where I would train an AI to play a "simple" card game. I will be doing this using the DQN with replay memory. The problem is, I can'...
3
votes
1answer
52 views

How to handle the final state in experience replay?

I'm using the DQN algorithm to train my agent to play a turn-based game. The memory replay buffer stores tuples of experiences $(s, a, r, s')$, where $s$ and $s'$ are consecutive states. At the last ...
3
votes
1answer
53 views

Can experience replay be used for training after completing every single epoch?

The DQN implements replay memory. Based on my research, I believe the replay memory starts to get used for training once there is enough experience in the memory buffer. This means the neural network ...
3
votes
1answer
43 views

How important is the choice of the initial state?

Is it crucial to always have the same initial (starting) state for Reinforcement Learning, for example, for Q-learning or DQN? Or it can vary?
3
votes
1answer
44 views

How do I represent a multi-dimensional state using a neural network?

I have a set of 15 unique playing cards from a deck of 52 playing cards. A given state is represented by the respective card values in the set of 15 cards, where the card value is a prime number ...
3
votes
1answer
40 views

Improving DQN with fluctuations

Hello :) I'm pretty new to this community, so let me know if I posted anything incorrectly and I'll try to change it. I'm working on the project which aim is to create self-driving agent in CARLA. I ...
3
votes
1answer
35 views

Is there a logical method of deducing an optimal batch size when training a Deep Q-learning agent with experience replay?

I am training an RL agent using Deep-Q learning with experience replay. At each frame, I am currently sampling 32 random transitions from a queue which stores a maximum of 20000 and training as ...
3
votes
1answer
58 views

How should I choose the target's update frequency in DQN?

I have been dealing with a problem that I'm trying to solve with DQN. A general question that I have is regarding the target's update frequency. How should it change? Depending on what factor do we ...
3
votes
1answer
58 views

How to know if my DQN is optimized?

I made a DQN that controls a traffic light. The observation states are the number of vehicles of each lane in the intersection. I trained it for 500 episodes and saved the model every 50th episode. I ...
3
votes
1answer
48 views

What's the right way of building a deep Q-network?

I'm new to RL and to deep q-learning and I have a simple question about the architecture of the neural network to use in an environment with a continous state space a discrete action space. I tought ...
3
votes
1answer
43 views

If agent chooses an action that the environment can't operate, how should I handle this situation?

I'm building a really simple experiment, letting an agent move from the bottom-left corner to the upper-right corner on a 3x3 squared paper. I plan to use DQN to do this. I'm having trouble handling ...
3
votes
1answer
42 views

Is a state that includes only the past n-step price records partially observable?

I'm currently working on a project to make an DQN agent that decides whether to charge or discharge an electric vehicle according to hourly changing price to sell or buy. The price pattern also varies ...
3
votes
1answer
94 views

Experience Replay Not Always Giving Better Results

I have recently started working on a control problem using a Deep Q Network as proposed by DeepMind (https://arxiv.org/abs/1312.5602). Initially, I implemented it without Experience Replay. The ...
3
votes
1answer
302 views

What are the differences between the DQN variants?

There are several variants of the DQN model. For example, double DQN, duelling DQN, prioritized DQN, distributed prioritized DQN, episodic memory DQN, asynchronous n-step DQN and multiple DQN. What ...
3
votes
1answer
165 views

Reason for issues with correlation in the dataset in DQN

From the paper Human level Control through DeepRL, the correlation in the data causes instability in the network and may causes the network to ...
3
votes
1answer
744 views

Why use semi-gradient instead of full gradient in RL problems, when using function approximation?

Semi-gradient methods work well in reinforcement learning, but what is there a reason of not using the true gradient if it can be computed? I tried it on the cart pole problem with a deep Q-network ...
3
votes
0answers
78 views

How to take actions at each episode and within each step of the episode in deep Q learning?

In deep Q learning, we execute the algorithm for each episode, and for each step within an episode, we take an action and record a reward. I have a situation where my action is 2-tuple $a=(a_1,a_2)$. ...
3
votes
0answers
43 views

Representation of state space, action space and reward system for Reinforcement Learning problem

I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without ...
3
votes
1answer
54 views

How does the optimization process in hindsight experience replay exactly work?

I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the ...
3
votes
0answers
56 views

DQN, how to choose the reward fucntion?

I built a simple AI system that tries to solve the 8 puzzle using DQN. The problem is, if the agent gets only a reward greater than zero when winning, the training will take a long time, so I made a ...
3
votes
0answers
43 views

What is the difference between random and sequential sampling from the reply memory?

I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure. ...
3
votes
0answers
29 views

Should importance sample weighting be compensated for by dynamically increasing learning rate?

I'm using Prioritized Experience Replay (PER) with a DDQN. To compensate for overfitting relatively high-value samples due to the non-uniform selection, I'm training with sample weights provided along ...