Questions tagged [dqn]

For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.

Filter by
Sorted by
Tagged with
12
votes
1answer
2k views

Why does DQN require two different networks?

I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...
7
votes
3answers
1k views

Huge action space size in Reinforcement Learning

I am working on creating a RL based AI for a certain board game. Just as a general overview of the game so that you understand what it's all about: It's a discrete turn-based game with a board of size ...
5
votes
2answers
704 views

What is the difference between DQN and AlphaGo Zero?

I have already implemented a relatively simple DQN on Pacman. Now I would like to clearly understand the difference between a DQN and the techniques used by AlphaGo zero/AlphaZero and I couldn't find ...
5
votes
2answers
262 views

Deciding on a reward per each action in a given state (Q-learning)

I looked for existing posts on Stack Exchange, which kind of answer the questions about the reward system and reward function, but not specifically what I want to ask here, which is how do you ...
5
votes
1answer
69 views

Can exogenous variables be state features in reinforcement learning?

I have a question about state representation of Q-learning or DQN algorithm. I'm still a beginner of RL, so I'm not sure that is it suitable to take exogenous variables as state features. For example,...
5
votes
2answers
241 views

Why don't people use projected Bellman error with deep neural networks?

Projected Bellman error has shown to be stable with linear function approximation. The technique is not at all new. I can only wonder why this technique is not adopted to use with non-linear function ...
5
votes
2answers
187 views

Can DQN perform better than Double DQN?

I'm training both kind of agents against an environment but DQN performs significantly better than Double DQN. As I've saw here, Double DQN use to perform better than DQN. Am I doing something wrong ...
5
votes
0answers
70 views

Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?

While reading the DQN paper, I found that randomly selecting and learning samples reduced divergence in RL using a non-linear function approximator (e.g a neural network). So why does Reinforcement ...
4
votes
3answers
245 views

What is the target Q-value in DQNs?

I understand that in DQNs, the loss is measured by taking the MSE of outputted Q-values and target Q-values. Whats does the target Q-values represent? And how is it obtained/calculated by the DQN?
4
votes
1answer
709 views

Ensure convergence of DDQN if true Q-values are very close

I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future reward from ...
4
votes
1answer
260 views

Which kind of prioritized experience replay should I use?

The Prioritized Experience Replay paper gives two different ways of sampling from the replay buffer. One, called "proportional prioritization", assigns each transition a priority proportional to its ...
4
votes
1answer
182 views

DQN in stochastic environment

I'm trying to apply a DQN to a stochastic environment but i'm having trouble getting it to converge. I found some similar questions asked here, but no solutions yet. I can fairly easy get the DQN to ...
4
votes
1answer
49 views

How can a DQN backpropagate its loss?

I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I ...
4
votes
1answer
48 views

How to represent players in a multi agent environment so each model can distinguish its own player

So I have 2 models trained with DQN algorithm that I want to train in a multi-agent environment to see how they react with each other. The models were trained in an environment consisting of 0's and 1'...
4
votes
0answers
241 views

What could be causing the drastic performance drop of the DQN model on the Pong environment?

I am running a basic DQN (Deep Q-Network) on the Pong environment. Not a CNN, just a 3 layer linear neural net with ReLUs. It seems to work for the most part, but at some point, my model suffers from ...
3
votes
2answers
726 views

Each training run for DDQN agent takes 2 days, and still ends up with -13 avg score, but OpenAi baseline DQN needs only an hour to converge to +18?

Status: For a few weeks now, I have been working on a Double DQN agent for the PongDeterministic-v4 environment, which you can find here. A single training run ...
3
votes
2answers
100 views

What are some online courses for deep reinforcement learning?

What are some (good) online courses for deep reinforcement learning? I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
3
votes
1answer
74 views

Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?

In the original prioritized experience replay paper, the authors track $\gamma_t$ in every state transition tuple (see line 6 in algorithm below): Why do the authors track this at every time step? ...
3
votes
1answer
48 views

How would researchers determine the best deep learning model if every run of the code yields different results?

There are many factors that cause the results of ML models to be different for every run of the same piece of code. One factor could be different initialization of weights in the neural network. ...
3
votes
1answer
53 views

Are Q values estimated from a DQN different from a duelling DQN with the same number of layers and filters?

I am confused about the Q values of a duelling deep Q network (DQN). As far as I know, duelling DQNs have 2 outputs Advantage: how good it is to be in a particular state $s$ Value: the advantage of ...
3
votes
2answers
678 views

My DQN is stuck and can't see where the problem is

I'm trying to replicate the DeepMind paper results, so I implemented my own DQN. I left it training for more than 4 million frames (more than 2000 episodes) on SpaceInvaders-v4 (OpenAI-Gym) and it ...
3
votes
1answer
950 views

DQN input representation for a card game

In order to learn about DP and RL, I chose to start a side project where I would train an AI to play a "simple" card game. I will be doing this using the DQN with replay memory. The problem is, I can'...
3
votes
1answer
40 views

Can experience replay be used for training after completing every single epoch?

The DQN implements replay memory. Based on my research, I believe the replay memory starts to get used for training once there is enough experience in the memory buffer. This means the neural network ...
3
votes
1answer
39 views

How important is the choice of the initial state?

Is it crucial to always have the same initial (starting) state for Reinforcement Learning, for example, for Q-learning or DQN? Or it can vary?
3
votes
1answer
39 views

How do I represent a multi-dimensional state using a neural network?

I have a set of 15 unique playing cards from a deck of 52 playing cards. A given state is represented by the respective card values in the set of 15 cards, where the card value is a prime number ...
3
votes
1answer
31 views

If agent chooses an action that the environment can't operate, how should I handle this situation?

I'm building a really simple experiment, letting an agent move from the bottom-left corner to the upper-right corner on a 3x3 squared paper. I plan to use DQN to do this. I'm having trouble handling ...
3
votes
1answer
280 views

How is the gradient of the loss function in DQN derived?

In the original DQN paper, page 1, the loss function of the DQN is $$ L_{i}(\theta_{i}) = \mathbb{E}_{(s,a,r,s') \sim U(D)} [(r+\gamma \max_{a'} Q(s',a',\theta_{i}^{-}) - Q(s,a;\theta_{i}))^2] $$ ...
3
votes
1answer
41 views

Is a state that includes only the past n-step price records partially observable?

I'm currently working on a project to make an DQN agent that decides whether to charge or discharge an electric vehicle according to hourly changing price to sell or buy. The price pattern also varies ...
3
votes
1answer
84 views

Experience Replay Not Always Giving Better Results

I have recently started working on a control problem using a Deep Q Network as proposed by DeepMind (https://arxiv.org/abs/1312.5602). Initially, I implemented it without Experience Replay. The ...
3
votes
1answer
218 views

What are the differences between the DQN variants?

There are several variants of the DQN model. For example, double DQN, duelling DQN, prioritized DQN, distributed prioritized DQN, episodic memory DQN, asynchronous n-step DQN and multiple DQN. What ...
3
votes
1answer
127 views

Reason for issues with correlation in the dataset in DQN

From the paper Human level Control through DeepRL, the correlation in the data causes instability in the network and may causes the network to ...
3
votes
0answers
31 views

Representation of state space, action space and reward system for Reinforcement Learning problem

I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without ...
3
votes
0answers
45 views

DQN, how to choose the reward fucntion?

I built a simple AI system that tries to solve the 8 puzzle using DQN. The problem is, if the agent gets only a reward greater than zero when winning, the training will take a long time, so I made a ...
3
votes
0answers
37 views

What is the difference between random and sequential sampling from the reply memory?

I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure. ...
3
votes
0answers
28 views

Should importance sample weighting be compensated for by dynamically increasing learning rate?

I'm using Prioritized Experience Replay (PER) with a DDQN. To compensate for overfitting relatively high-value samples due to the non-uniform selection, I'm training with sample weights provided along ...
3
votes
0answers
138 views

Deep Q-Network (DQN) to learn the game 2048

I am trying to build a Deep Q-Network (DQN) agent that can learn to play the game 2048. I am orientating myself on other programs and articles that are based on the game snake and it worked well (...
3
votes
0answers
107 views

DQN Agent not learning anymore - what can I do to fix this?

I am trying to use Deep-Q-Learning to learn an ANN which controls a 7-DOF robotic arm. The robotic arm must avoid an obstacle and reach a target. I have implemented a number of state-of-art ...
3
votes
0answers
142 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
3
votes
1answer
77 views

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

Twin Delayed Deep Deterministic (TD3) policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained ...
2
votes
1answer
1k views

Is Deep Q Neural Network (DQN) applicable only with images as inputs?

More precisely: is DQNN applicable only when we have high translational invariance in our input(s)? Starting from the original paper on nature (here a version stored on googleapis) and after looking ...
2
votes
1answer
96 views

Can DQN announce it has things in its hand in a card game?

More informations on the card game I'm talking about are in my last question here: DQN input representation for a card game So I was thinking about the output of the q neural network and, aside from ...
2
votes
2answers
151 views

What should the target be when the neural network outputs multiple Q values in deep Q-learning?

I have some gaps in my understanding regarding the performing of the gradient descent in Deep - Q networks. The original deep q network for Atari performs a gradient descent step to minimise $y_j - Q(...
2
votes
1answer
72 views

Can the opponent's turn affect the reward for a DQN agent action?

I made an engine for a 2 players card game and now I am trying to make an environment similar to OpenAI Gym envs, to ease out the training. I fail to understand this thing however: If I use ...
2
votes
1answer
74 views

Convergence in multi-agent environment

I have a multi-agent environment where agents are trying to optimise the overall energy consumption of their group. Agents can exchange energy between themselves (actions for exchange of energy ...
2
votes
1answer
42 views

How to evaluate a Deep Q-Network

Good day, it's a pleasure having joined this Stack. In my master thesis I have to expand a Deep Reinforcement Learning Network, to be precise a Deep Q-Network, which is used to control machines in an ...
2
votes
1answer
35 views

Intutitive explanation of why Experience Replay is used in a Deep Q Network?

I understand that Experience Replay is used for data efficiency reasons and to remove correlations in sequences of data. How exactly do these sequences of correlated data affect the performance of the ...
2
votes
1answer
67 views

How is the expected value in the loss function of DQN approximated?

In Deep Q Learning the parametrized Q-functions $Q_i$ are optimised by performing gradient descent on the series of loss functions $L_i(\theta_i)= E_{(s,a)\sim p}[(y_i-Q(s,a;\theta_i))^2]$ , where ...
2
votes
4answers
158 views

How to stop DQN Q function from increasing during learning?

Following the DQN algorithm with experience replay: We calculate the $loss=(Q(s,a)-(r+Q(s+1,a)))^2$. Assume I have positive but changing rewards. Meaning, $r>0$. Thus, since the rewards are ...
2
votes
1answer
27 views

Why do DQNs use linear activations on cartpole?

I've been reading a lot of tutorials on DQNs for cartpole. In many of them, they have the funnel layer of the neural net be a linear activation. Why is this? Is it just a choice made by the ...
2
votes
2answers
377 views

What does it mean by high dimensional state in DQN?

Going through the DQN paper, it said the state-space is high dimensional. I am a little bit confused here. Suppose my state is a high dimensional vector of N length ...