Questions tagged [dqn]

For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.

Filter by
Sorted by
Tagged with
3
votes
1answer
68 views

How to handle changing goals in a DQN?

I created a virtual 2D environment where an agent aims to find a correct pose corresponding to a target image. I implemented a DQN to solve this task. When the goal is fixed, e.g. the aim is to find ...
3
votes
1answer
60 views

How to know if my DQN is optimized?

I made a DQN that controls a traffic light. The observation states are the number of vehicles of each lane in the intersection. I trained it for 500 episodes and saved the model every 50th episode. I ...
3
votes
1answer
63 views

What's the right way of building a deep Q-network?

I'm new to RL and to deep q-learning and I have a simple question about the architecture of the neural network to use in an environment with a continous state space a discrete action space. I tought ...
3
votes
1answer
43 views

Is a state that includes only the past n-step price records partially observable?

I'm currently working on a project to make an DQN agent that decides whether to charge or discharge an electric vehicle according to hourly changing price to sell or buy. The price pattern also varies ...
3
votes
1answer
104 views

Experience Replay Not Always Giving Better Results

I have recently started working on a control problem using a Deep Q Network as proposed by DeepMind (https://arxiv.org/abs/1312.5602). Initially, I implemented it without Experience Replay. The ...
3
votes
1answer
388 views

What are the differences between the DQN variants?

There are several variants of the DQN model. For example, double DQN, duelling DQN, prioritized DQN, distributed prioritized DQN, episodic memory DQN, asynchronous n-step DQN and multiple DQN. What ...
3
votes
1answer
179 views

Reason for issues with correlation in the dataset in DQN

From the paper Human level Control through DeepRL, the correlation in the data causes instability in the network and may causes the network to ...
3
votes
1answer
243 views

How to compute the action probabilities with Thompson sampling in deep Q-learning?

In some implementations of off-policy Q-learning, we need to know the action probabilities given by the behavior policy $\mu(a)$ (e.g., if we want to use importance sampling). In my case, I am using ...
3
votes
1answer
772 views

Why use semi-gradient instead of full gradient in RL problems, when using function approximation?

Semi-gradient methods work well in reinforcement learning, but what is there a reason of not using the true gradient if it can be computed? I tried it on the cart pole problem with a deep Q-network ...
3
votes
0answers
81 views

How to take actions at each episode and within each step of the episode in deep Q learning?

In deep Q learning, we execute the algorithm for each episode, and for each step within an episode, we take an action and record a reward. I have a situation where my action is 2-tuple $a=(a_1,a_2)$. ...
3
votes
0answers
39 views

Does the concept of validation loss apply to training deep Q networks?

In deep learning, the concept of validation loss is to ensure that the model being trained is not currently overfitting the data. Is there a similar concept of overfitting in deep q learning? Given ...
3
votes
0answers
43 views

Representation of state space, action space and reward system for Reinforcement Learning problem

I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without ...
3
votes
0answers
118 views

How to correctly implement self-play with DQN?

I have an environment where an agent faces an equal opponent, and while I've achieved OK performance implementing DQN and treating the opponent as a part of the environment, I think performance would ...
3
votes
0answers
52 views

Is this a good approach to solving Atari's “Montezuma's Revenge”?

I'm new to Reinforcement Learning. For an internship, I am currently training Atari's "Montezuma's Revenge" using a double Deep Q-Network with Hindsight Experience Replay (HER) (see also ...
3
votes
1answer
71 views

How does the optimization process in hindsight experience replay exactly work?

I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the ...
3
votes
0answers
60 views

DQN, how to choose the reward fucntion?

I built a simple AI system that tries to solve the 8 puzzle using DQN. The problem is, if the agent gets only a reward greater than zero when winning, the training will take a long time, so I made a ...
3
votes
0answers
46 views

What is the difference between random and sequential sampling from the reply memory?

I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure. ...
3
votes
0answers
29 views

Should importance sample weighting be compensated for by dynamically increasing learning rate?

I'm using Prioritized Experience Replay (PER) with a DDQN. To compensate for overfitting relatively high-value samples due to the non-uniform selection, I'm training with sample weights provided along ...
3
votes
0answers
165 views

Deep Q-Network (DQN) to learn the game 2048

I am trying to build a Deep Q-Network (DQN) agent that can learn to play the game 2048. I am orientating myself on other programs and articles that are based on the game snake and it worked well (...
3
votes
0answers
178 views

DQN Agent not learning anymore - what can I do to fix this?

I am trying to use Deep-Q-Learning to learn an ANN which controls a 7-DOF robotic arm. The robotic arm must avoid an obstacle and reach a target. I have implemented a number of state-of-art ...
3
votes
0answers
168 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
3
votes
1answer
110 views

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

Twin Delayed Deep Deterministic (TD3) policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained ...
2
votes
1answer
2k views

Is Deep Q Neural Network (DQN) applicable only with images as inputs?

More precisely: is DQNN applicable only when we have high translational invariance in our input(s)? Starting from the original paper on nature (here a version stored on googleapis) and after looking ...
2
votes
1answer
114 views

Can DQN announce it has things in its hand in a card game?

More informations on the card game I'm talking about are in my last question here: DQN input representation for a card game So I was thinking about the output of the q neural network and, aside from ...
2
votes
1answer
140 views

How should I compute the target for updating in a DQN at the terminal state if I have pseudo-episodes?

I'm training a DQN in a real environment where I do not have a natural terminal state, so I've built the episode in an artificial way (i.e. it starts in a random condition and after T steps it ends). ...
2
votes
1answer
69 views

Reinforcement learning with action consisting of two discrete values

I'm new to reinforcement learning. I have a problem where an action is composed of an order (rod with a required length) and an item from a warehouse (an existing rod with a certain length, which will ...
2
votes
1answer
63 views

Should illegal moves be excluded from loss calculation in DQN algorithm?

I'm implementing DQN algorithm to train my agent to play a turn-based game. The action space for the game is small, but not all moves are available at all the states. Therefore, when deciding on which ...
2
votes
2answers
236 views

What should the target be when the neural network outputs multiple Q values in deep Q-learning?

I have some gaps in my understanding regarding the performing of the gradient descent in Deep - Q networks. The original deep q network for Atari performs a gradient descent step to minimise $y_j - Q(...
2
votes
1answer
69 views

When training a DQN, how should we update the value of actions that were not taken?

Let's say that we have three actions. The highest-valued action of the three choices is the first. When training the DQN, what do we do with the other two, as we don't have a target for them, since ...
2
votes
1answer
102 views

Can the opponent's turn affect the reward for a DQN agent action?

I made an engine for a 2 players card game and now I am trying to make an environment similar to OpenAI Gym envs, to ease out the training. I fail to understand this thing however: If I use ...
2
votes
1answer
96 views

Convergence in multi-agent environment

I have a multi-agent environment where agents are trying to optimise the overall energy consumption of their group. Agents can exchange energy between themselves (actions for exchange of energy ...
2
votes
2answers
95 views

How should I define the reward function to solve the Wumpus game with deep Q-learning?

I'm writing a DQN agent for the Wumpus game. Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...
2
votes
1answer
26 views

How is weighted average computed in Deep Q networks

I was going through the Sutton book and they said the update formula for Q learning comes from the weighted average of the returns I.e New estimate= old estimate +alpha*[returns- old estimate] So by ...
2
votes
1answer
104 views

How to evaluate a Deep Q-Network

Good day, it's a pleasure having joined this Stack. In my master thesis I have to expand a Deep Reinforcement Learning Network, to be precise a Deep Q-Network, which is used to control machines in an ...
2
votes
1answer
67 views

Why can't DQN be used for self-driving cars?

Why can't DQN be used for self-driving cars? Why can't DQN and similar RL algorithms be used for self-driving cars? The reason why I am curious is that it successfully plays go and other multistate ...
2
votes
1answer
44 views

Intutitive explanation of why Experience Replay is used in a Deep Q Network?

I understand that Experience Replay is used for data efficiency reasons and to remove correlations in sequences of data. How exactly do these sequences of correlated data affect the performance of the ...
2
votes
1answer
121 views

How is the expected value in the loss function of DQN approximated?

In Deep Q Learning the parametrized Q-functions $Q_i$ are optimised by performing gradient descent on the series of loss functions $L_i(\theta_i)= E_{(s,a)\sim p}[(y_i-Q(s,a;\theta_i))^2]$ , where ...
2
votes
1answer
206 views

How do we compute the target value when the agent ends up in the terminal state?

I am working on a deep reinforcement learning problem. Throughout the episode, there is a small positive and negative reward for good or bad decisions. In the end, there is a huge reward for the ...
2
votes
1answer
35 views

Why do DQNs use linear activations on cartpole?

I've been reading a lot of tutorials on DQNs for cartpole. In many of them, they have the funnel layer of the neural net be a linear activation. Why is this? Is it just a choice made by the ...
2
votes
1answer
596 views

DQN Breakout adding an extra negative reward to help training?

I'm trying to train a DQN, so I'm using OpenAI gym and Breakout (Breakout-v0). I have altered the reward supplied by the environment: If the episode is not completed fully, the agent gets a -10 ...
2
votes
1answer
57 views

How can I model a problem as an MDP if the agent does not follow the successive order of states?

In my problem, the agent does not follow the successive order of states, but selects with $\epsilon$-greedy the best pair (state, action) from a priority queue. More specifically, when my agent goes ...
2
votes
1answer
60 views

Keras DQN Model with Multiple Inputs and Multiple Outputs [closed]

I am trying to create a DQN agent where I have 2 inputs: the agent's position and a matrix of 0s and 1s. The output is composed of the agent's new chosen position, a matrix of 0s and 1s (different ...
2
votes
1answer
43 views

What is the optimal exploration-exploitation trade-off in Q*bert?

I am training an RL agent with Deep Q-learning + Experience Replay on the Q*bert Atari environment. After 400,000 frames, my agent appears to have learned strategic information about the game, but ...
2
votes
1answer
56 views

When using experience replay in reinforcement learning, which state is used for training?

I'm slightly confused about the experience replay process. I understand why we use batch processing in reinforcement learning, and from my understanding, a batch of states is input into the neural ...
2
votes
1answer
161 views

What exactly is the advantage of double DQN over DQN?

I started looking into the double DQN (DDQN). Apparently, the difference between DDQN and DQN is that in DDQN we use the main value network for action selection and the target network for outputting ...
2
votes
1answer
91 views

How does the target network in double DQNs find the maximum Q value for each action?

I understand the fact that the neural network is used to take the states as inputs and it outputs the Q-value for state-action pairs. However, in order to compute this and update its weights, we need ...
2
votes
1answer
54 views

Why do some DQN implementations not require random exploration but instead emulate all actions?

I've found online some DQN algorithms that (in a problem with a continuous state space and few actions, let's say 2 or 3), at each time step, compute and store (in the memory used for updating) all ...
2
votes
1answer
46 views

Should the agent play the game until the end or until the winner is found?

I'm using the DQN algorithm to train my agent to play a turn-based game. The winner of the game can be known before the game is over. Once the winning condition is satisfied, it cannot be reverted. ...
2
votes
1answer
51 views

Will the target network, which is less trained than the normal network, output inferior estimates?

I'm having some trouble understanding some parts of the usage of target networks. I get that having the same network predict the state/action/advantage values for both the current networks can lead ...
2
votes
1answer
2k views

In DQN, updating target network every N steps or slowly update every step is better?

The use of target network is to reduce the chance of value divergence which could happen with off-policy samples trained with semi-gradient objectives. In Deep Q network, semi-gradient TD is used and ...