Questions tagged [dqn]

For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.

Filter by
Sorted by
Tagged with
3
votes
1answer
39 views

Improving DQN with fluctuations

Hello :) I'm pretty new to this community, so let me know if I posted anything incorrectly and I'll try to change it. I'm working on the project which aim is to create self-driving agent in CARLA. I ...
2
votes
2answers
37 views

How should I define the reward function to solve the Wumpus game with deep Q-learning?

I'm writing a DQN agent for the Wumpus game. Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...
2
votes
0answers
34 views

How to model this problem to solve using DQN?

I have a scheduling problem as follows. At each time $t=1,2,\ldots,T$, a set of $n$ jobs arrives where job $j$ has a cost $c_{j,t}$ and a budget $b_{j,t}$ which are revealed to the decision maker for ...
1
vote
1answer
69 views

Reinforcement learning simple problem: agent not learning, wrong action

I am pretty new to RL and I am trying to code a simple RL task with pytorch. The goal/task is the following: The initial state is $t_o$ and the agent takes an action $\Delta_t$: $t_o +\Delta_t = t_1$. ...
0
votes
0answers
22 views

Pytorch and keras ddqn seem identical, only keras learns

I followed a tutorial for ddqn to beat pong, it beats it with a perfect score in keras, but trying to translate it to pytorch it doesn't learn at all. What am I missing? I pasted all the code for each ...
1
vote
1answer
47 views

Why do my rewards reduce after extensive training using D3QN?

I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually ...
1
vote
2answers
77 views

How does one know that a problem is “model-free” in reinforcement learning?

Consider this slide from a Stanford lecture on reinforcement learning. It states that a model is the agent's representation of how the world changes in response to the agent's action. I've been ...
0
votes
0answers
19 views

Reward and loss follow the same shape in DQN

If the accumulated reward increases, the loss increases and vice versa. This is a strange behaviour. See the figure below for an example. What is the possibility of having this behaviour in DQN? I ...
1
vote
0answers
20 views

Is it feasible to train a DQN with thousands of input ports?

I designed a DQN architecture for some problem. The problem has a parameter $m$ as the number of clients. In my situation, $m$ is large, $m\in\{100,200,\ldots,1000\}$. For this situation, the number ...
1
vote
1answer
39 views

What is the optimal exploration-exploitation trade-off in Q*bert?

I am training an RL agent with Deep Q-learning + Experience Replay on the Q*bert Atari environment. After 400,000 frames, my agent appears to have learned strategic information about the game, but ...
0
votes
1answer
27 views

How to calculate v min and v max for C51 DQN

Background: In C51 DQNs you must specify a v-min/max to be used during training. The way this is generally done is you take the max score possible for the game and set that to v-max, then v-min is ...
3
votes
1answer
35 views

Is there a logical method of deducing an optimal batch size when training a Deep Q-learning agent with experience replay?

I am training an RL agent using Deep-Q learning with experience replay. At each frame, I am currently sampling 32 random transitions from a queue which stores a maximum of 20000 and training as ...
0
votes
0answers
17 views

How to train a hierarchical DQN to play the Montezuma's Revenge game?

Would anybody share the experience on how to train a hierarchical DQN to play the Montezuma's Revenge game? How should I design the reward function? How should I balance the anneal rate of the two-...
1
vote
0answers
44 views

Why scaling reward drastically affects performance?

I have devised an gridworld-like environment where a RL agent is tasked to cover all the blank squares by passing through them. Possible actions are up, down, left, right. The reward scheme is the ...
1
vote
1answer
52 views

What are the variables that need to be saved and loaded, so that a DQN model starts where it left off?

TensorFlow allows users to save the weights and the model architecture, however, that will be insufficient unless the values of certain other variables are also stored. For instance, in DQN, if $\...
1
vote
1answer
46 views

How is exponential moving average computed in deep Q networks?

In normal Q-learning, the update rule is an implementation of the exponential moving average, which then converges to the optimal true Q values. However, looking at DQN, how exactly is the exponential ...
2
votes
0answers
24 views

Combine DQN with the Average Reward setting

I have to deal with a non-episodic task, where there is addittionally a continuous state space and more specifically in each time step there is always a new state that has never been seen before. I ...
1
vote
1answer
23 views

How is weighted average computed in Deep Q networks

I was going through the Sutton book and they said the update formula for Q learning comes from the weighted average of the returns I.e New estimate= old estimate +alpha*[returns- old estimate] So by ...
2
votes
1answer
112 views

How should I compute the target for updating in a DQN at the terminal state if I have pseudo-episodes?

I'm training a DQN in a real environment where I do not have a natural terminal state, so I've built the episode in an artificial way (i.e. it starts in a random condition and after T steps it ends). ...
3
votes
1answer
58 views

How should I choose the target's update frequency in DQN?

I have been dealing with a problem that I'm trying to solve with DQN. A general question that I have is regarding the target's update frequency. How should it change? Depending on what factor do we ...
2
votes
1answer
70 views

How to compute the target for double Q-learning update step?

I've already read the original paper about double DQN but I do not find a clear and practical explanation of how the target $y$ is computed, so here's how I interpreted the method (let's say I have 3 ...
1
vote
1answer
35 views

In DQN, when do the parameters in the Neural Network update based on the reward received?

I'm aware that we back-propagate after computing the loss between: The Neural Network Q values and the Target Network Q values However, all this is doing is updating the parameters of the Neural ...
1
vote
1answer
30 views

Why are Target Networks used in Deep Q-Learning as opposed to the Expected Value equation?

I understand we use a target network because it helps resolve issues regarding stability, however, that's not what I'm here to ask. What I would like to understand is why a target network is used as a ...
2
votes
1answer
48 views

When using experience replay in reinforcement learning, which state is used for training?

I'm slightly confused about the experience replay process. I understand why we use batch processing in reinforcement learning, and from my understanding, a batch of states is input into the neural ...
4
votes
2answers
113 views

My Deep Q-Learning Network does not learn for OpenAI gym's cartpole problem

I am implementing OpenAI gym's cartpole problem using Deep Q-Learning (DQN). I followed tutorials (video and otherwise) and learned all about it. I implemented a code for myself and I thought it ...
0
votes
0answers
33 views

How does DQN convergence work in reinforcement learning

In supervised learning we have an unbiased target value, but in reinforcement learning this isn’t the case The network predicts its own target value, now how exactly does it converge if the network ...
0
votes
0answers
31 views

Why do we use the Target Network for action evaluation in Double deep Q networks

Is there any specific reason as to why The target Network is used for evaluation and The online network Is used for selection, what would be the difference if both roles were switched, our online ...
1
vote
0answers
32 views

How can deep Q-learning converge if the targets may not be correct?

In deep Q-learning, $Q(s, a)$ and $Q'(s, a)$ are predicted or estimated by the neural network itself. In supervised learning, the target value is a true unbiased value. However, this isn't the case in ...
0
votes
0answers
38 views

Having trouble understanding how Double deep Q networks work

I’ve looked at various articles and I’m still very confused, I understand the normal double Q learning about having two Action value estimates that use two different set of samples But coming to ...
1
vote
0answers
69 views

What happens if our target network overestimates the value?

When we use DDQN, we often use the target network in case our online network overestimates a value, but this doesn't make sense to me, because What happens if our target network is the one that ...
1
vote
1answer
94 views

What exactly is the advantage of DDQN over DQN

I started looking into DDQN and apparently the difference is we use our Online network for action selection, And we use our target network for outputting the Q values, I don’t quite get how this is ...
0
votes
0answers
54 views

Why does my agent always takes a same action in RL?

I'm trying to reproduce the work in the paper Demand Response for Home Energy Management Using Reinforcement Learning and Artificial Neural Network. I want to optimize the power consumption for home ...
4
votes
1answer
180 views

Why do DQNs tend to forget?

Why do DQNs tend to forget? Is it because when you feed highly correlated samples, your model (function approximation) doesn't give a general solution? For example: I use level 1 experiences, my ...
0
votes
0answers
41 views

How can I build a deep reinforcement learning model that can be trained with multiple time series datasets

I built a DRL model to trade stocks in the financial market but the number of observations is relatively small and I would like to increase it by training the same model with stocks from several ...
2
votes
1answer
68 views

Reinforcement learning with action consisting of two discrete values

I'm new to reinforcement learning. I have a problem where an action is composed of an order (rod with a required length) and an item from a warehouse (an existing rod with a certain length, which will ...
2
votes
1answer
76 views

How does the target network in double DQNs find the maximum Q* value for each action?

I understand the fact that the neural network is used to take the states as inputs and it outputs the Q-value for state-action pairs. However, in order to compute this and update its weights, we need ...
0
votes
0answers
35 views

How to define image as observation in tensorflow_agent?

I want to use the DQN algorithm from TensorFlow Agent API for a custom environment. In my environment, the agent's observations are image sequences. As far as I know, images should be encoded by a ...
4
votes
3answers
266 views

Upper limit to the maximum cumulative reward in a deep reinforcement learning problem

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem? For example you want to train an DQN agent in an environment and you want to know what is the highest ...
2
votes
0answers
50 views

Prioritised Remembering in Experience Replay (Q-Learning)

I'm using Experience Replay based on the original Prioritized Experience Replay (PER) paper. In the paper authors show ~ an order of magnitude increase in data efficiency from prioritized sampling. ...
1
vote
1answer
62 views

Why do we need target network in deep Q learning? [duplicate]

I already know deep RL, but to learn it deeply I want to know why do we need 2 networks in deep RL. What does the target network do? I now there is huge mathematics into this, but I want to know deep ...
1
vote
1answer
52 views

Why does adding another network help in double DQN?

What is the idea behind double DQN? The Bellman equation used to calculate the Q values to update the online network follows the equation: ...
0
votes
1answer
48 views

How does training for DQN work if messing up in the environment in costly?

Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works. Is it that ...
2
votes
1answer
52 views

Why do some DQN implementations not require random exploration but instead emulate all actions?

I've found online some DQN algorithms that (in a problem with a continuous state space and few actions, let's say 2 or 3), at each time step, compute and store (in the memory used for updating) all ...
0
votes
0answers
41 views

What does the notation “for t=T to 1,−1 do” in terms of time steps, in deep recurrent q network?

In looking at an algorithm in the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning. Here is the full algorithm: What does the notation ...
1
vote
0answers
31 views

What is the role of embeddings in a deep recurrent Q network?

When describing the model architecture for a deep recurrent q network, the authors of the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning each agent consists of a recurrent ...
0
votes
0answers
31 views

DQN Tic-Tac-Toe does not quite become optimal

I am trying to implement a DQN agent for playing standard 3x3 Tic-Tac-Toe (it is a double DQN with experience replay, and using a target network). I got the hyperparameters to the point where the ...
0
votes
1answer
46 views

What reinforcement learning algorithm should I use in continuous states?

I want to use reinforcement learning in an environment I made. The exact environment doesn't really matter, but it comes down to this: The amount of different states in the environment is infinite e.g....
2
votes
1answer
55 views

Why does shifting all the rewards have a different impact on the performance of the agent?

I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
2
votes
1answer
38 views

Should the agent play the game until the end or until the winner is found?

I'm using the DQN algorithm to train my agent to play a turn-based game. The winner of the game can be known before the game is over. Once the winning condition is satisfied, it cannot be reverted. ...
1
vote
0answers
52 views

Atari Games: Pretrained CNN to accelerate training?

DQN for Atari takes considerable training time. For example, the 2015 paper in Nature notes that algorithms are trained for 50 million frames or equivalently around 38 days of game experience in total....