Questions tagged [dqn]

For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.

Filter by
Sorted by
Tagged with
0
votes
0answers
29 views

How does DQN convergence work in reinforcement learning

In supervised learning we have an unbiased target value, but in reinforcement learning this isn’t the case The network predicts its own target value, now how exactly does it converge if the network ...
0
votes
0answers
30 views

Why do we use the Target Network for action evaluation in Double deep Q networks

Is there any specific reason as to why The target Network is used for evaluation and The online network Is used for selection, what would be the difference if both roles were switched, our online ...
1
vote
0answers
31 views

How can deep Q-learning converge if the targets may not be correct?

In deep Q-learning, $Q(s, a)$ and $Q'(s, a)$ are predicted or estimated by the neural network itself. In supervised learning, the target value is a true unbiased value. However, this isn't the case in ...
0
votes
0answers
36 views

Having trouble understanding how Double deep Q networks work

I’ve looked at various articles and I’m still very confused, I understand the normal double Q learning about having two Action value estimates that use two different set of samples But coming to ...
2
votes
1answer
50 views

Why does shifting all the rewards have a different impact on the performance of the agent?

I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
1
vote
1answer
82 views

What exactly is the advantage of DDQN over DQN

I started looking into DDQN and apparently the difference is we use our Online network for action selection, And we use our target network for outputting the Q values, I don’t quite get how this is ...
-2
votes
0answers
48 views

Taking the mean as an estimate in Double Q learning

enter image description here My question is , from the images above you can see there was a normal distribution of the rewards when going left , I do not understand why they took the average of the ...
1
vote
0answers
66 views

What happens if our target network overestimates the value?

When we use DDQN, we often use the target network in case our online network overestimates a value, but this doesn't make sense to me, because What happens if our target network is the one that ...
0
votes
0answers
49 views

Why does my agent always takes a same action in RL?

I'm trying to reproduce the work in the paper Demand Response for Home Energy Management Using Reinforcement Learning and Artificial Neural Network. I want to optimize the power consumption for home ...
4
votes
1answer
167 views

Why do DQNs tend to forget?

Why do DQNs tend to forget? Is it because when you feed highly correlated samples, your model (function approximation) doesn't give a general solution? For example: I use level 1 experiences, my ...
0
votes
0answers
40 views

How can I build a deep reinforcement learning model that can be trained with multiple time series datasets

I built a DRL model to trade stocks in the financial market but the number of observations is relatively small and I would like to increase it by training the same model with stocks from several ...
2
votes
1answer
67 views

Reinforcement learning with action consisting of two discrete values

I'm new to reinforcement learning. I have a problem where an action is composed of an order (rod with a required length) and an item from a warehouse (an existing rod with a certain length, which will ...
6
votes
1answer
90 views

Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?

While reading the DQN paper, I found that randomly selecting and learning samples reduced divergence in RL using a non-linear function approximator (e.g a neural network). So why does Reinforcement ...
2
votes
1answer
59 views

How does the target network in double DQNs find the maximum Q* value for each action?

I understand the fact that the neural network is used to take the states as inputs and it outputs the Q-value for state-action pairs. However, in order to compute this and update its weights, we need ...
1
vote
2answers
207 views

Understanding the loss function in deep Q-learning

I am trying to understand how deep Q learning (DQN) works. To my current understanding, each $Q(s, a)$ functions is estimated to be a function of a feature vector of its state $\phi$(s) and the weight ...
0
votes
0answers
35 views

How to define image as observation in tensorflow_agent?

I want to use the DQN algorithm from TensorFlow Agent API for a custom environment. In my environment, the agent's observations are image sequences. As far as I know, images should be encoded by a ...
1
vote
1answer
97 views

Why isn't my implementation of DQN using TensorFlow on the FrozenWorld environment working?

I am trying to test DQN on FrozenWorld environment in gym using TensorFlow 2.x. The update rule is (off policy) $$Q(s,a) \leftarrow Q(s,a)+\alpha (r+\gamma~ max_{a'}Q(s',a')-Q(s,a))$$ I am using an ...
4
votes
3answers
242 views

Upper limit to the maximum cumulative reward in a deep reinforcement learning problem

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem? For example you want to train an DQN agent in an environment and you want to know what is the highest ...
2
votes
0answers
49 views

Prioritised Remembering in Experience Replay (Q-Learning)

I'm using Experience Replay based on the original Prioritized Experience Replay (PER) paper. In the paper authors show ~ an order of magnitude increase in data efficiency from prioritized sampling. ...
2
votes
1answer
200 views

Deep Reinforcement Learning: Rewards suddenly dip down

I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using ...
1
vote
1answer
52 views

Why do we need target network in deep Q learning? [duplicate]

I already know deep RL, but to learn it deeply I want to know why do we need 2 networks in deep RL. What does the target network do? I now there is huge mathematics into this, but I want to know deep ...
1
vote
1answer
49 views

Why does adding another network help in double DQN?

What is the idea behind double DQN? The Bellman equation used to calculate the Q values to update the online network follows the equation: ...
1
vote
1answer
57 views

Do smaller loss values during DQN training produce better policies?

During the training of DQN, I noticed that the model with prioritized experience replay (PER) had a smaller loss in general compared to a DQN without PER. The mean squared loss was an order of ...
0
votes
1answer
46 views

How does training for DQN work if messing up in the environment in costly?

Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works. Is it that ...
4
votes
1answer
247 views

Why does the DQN not converge when the start or goal states can change dynamically?

I'm trying to apply a DQN to a stochastic environment, but I'm having trouble getting it to converge. I found some similar questions asked here, but no solutions yet. I can easily get the DQN to ...
2
votes
1answer
51 views

Why do some DQN implementations not require random exploration but instead emulate all actions?

I've found online some DQN algorithms that (in a problem with a continuous state space and few actions, let's say 2 or 3), at each time step, compute and store (in the memory used for updating) all ...
0
votes
0answers
41 views

What does the notation “for t=T to 1,−1 do” in terms of time steps, in deep recurrent q network?

In looking at an algorithm in the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning. Here is the full algorithm: What does the notation ...
1
vote
0answers
31 views

What is the role of embeddings in a deep recurrent Q network?

When describing the model architecture for a deep recurrent q network, the authors of the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning each agent consists of a recurrent ...
0
votes
1answer
44 views

What reinforcement learning algorithm should I use in continuous states?

I want to use reinforcement learning in an environment I made. The exact environment doesn't really matter, but it comes down to this: The amount of different states in the environment is infinite e.g....
0
votes
0answers
27 views

DQN Tic-Tac-Toe does not quite become optimal

I am trying to implement a DQN agent for playing standard 3x3 Tic-Tac-Toe (it is a double DQN with experience replay, and using a target network). I got the hyperparameters to the point where the ...
1
vote
0answers
45 views

Atari Games: Pretrained CNN to accelerate training?

DQN for Atari takes considerable training time. For example, the 2015 paper in Nature notes that algorithms are trained for 50 million frames or equivalently around 38 days of game experience in total....
0
votes
0answers
33 views

Why doesn't my double deep Q network trained with the same training set give consistent performance?

I've written a Double DQN which can do either 1-step or multi-step learning. It also has a prioritised reply buffer. The internal network is an LSTM. My input data is a series of time series data and ...
2
votes
1answer
37 views

Should the agent play the game until the end or until the winner is found?

I'm using the DQN algorithm to train my agent to play a turn-based game. The winner of the game can be known before the game is over. Once the winning condition is satisfied, it cannot be reverted. ...
3
votes
2answers
144 views

What are some online courses for deep reinforcement learning?

What are some (good) online courses for deep reinforcement learning? I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
4
votes
1answer
54 views

How to represent players in a multi agent environment so each model can distinguish its own player

So I have 2 models trained with the DQN algorithm that I want to train in a multi-agent environment to see how they react with each other. The models were trained in an environment consisting of 0's ...
2
votes
1answer
40 views

Should illegal moves be excluded from loss calculation in DQN algorithm?

I'm implementing DQN algorithm to train my agent to play a turn-based game. The action space for the game is small, but not all moves are available at all the states. Therefore, when deciding on which ...
0
votes
1answer
43 views

Why do we update the weights of the target network in deep Q learning?

I know we keep the target network constant during training to improve stability, but why exactly are we updating the weights of our target network? In particular, if we've already reached convergence, ...
0
votes
0answers
43 views

DQN flappybird does not learn

I'm trying to implement DQN to play flappy bird. The action space contains two actions: flap and do nothing. The input for the network is four stacked $80 \times 80$ images. I've been playing with the ...
3
votes
1answer
45 views

How to handle the final state in experience replay?

I'm using the DQN algorithm to train my agent to play a turn-based game. The memory replay buffer stores tuples of experiences $(s, a, r, s')$, where $s$ and $s'$ are consecutive states. At the last ...
3
votes
1answer
47 views

What happens when you select actions using softmax instead of epsilon greedy in DQN?

I understand the two major branches of RL are Q-Learning and Policy Gradient methods. From my understanding (correct me if I'm wrong), policy gradient methods have an inherent exploration built-in as ...
2
votes
1answer
50 views

How to handle changing goals in a DQN?

I created a virtual 2D environment where an agent aims to find a correct pose corresponding to a target image. I implemented a DQN to solve this task. When the goal is fixed, e.g. the aim is to find ...
0
votes
1answer
38 views

How to validate that my DQN hyperparameters are the optimal?

My DQN model outputs the best traffic light state in an intersection. I used different values of batch size and learning rate to find the best model. How would I know if I got the optimal ...
1
vote
0answers
31 views

Is there a way to show convergence of DQN other than by eye observation?

I made a DQN model and plot its reward curve. You can see intuitively that the curve already converged since its reward value now just oscillates. How can I show confidence that my DQN already reached ...
3
votes
1answer
55 views

How to know if my DQN is optimized?

I made a DQN that controls a traffic light. The observation states are the number of vehicles of each lane in the intersection. I trained it for 500 episodes and saved the model every 50th episode. I ...
1
vote
0answers
35 views

Designing a reward function for my reinforcement learning problem

I'm working on a project lately and I'm trying to solve a problem with reinforcement learning and I have serious issues with shaping the reward function. The problem is designing a device with maximum ...
0
votes
1answer
62 views

How do I know that the DQN has learnt an appropriate Q function?

Is there any sanity check to know whether the Q functions learnt are appropriate in deep Q networks? I know that the Q values for end states should approximate the terminal reward. However, is it ...
2
votes
1answer
62 views

NoisyNet DQN with default parameters not exploring

I implemented a DQN algorithm that plays OpenAIs Cartpole environment. The NN architecture consists of 3 normal linear layers that encode the state, and one noisy linear layer, that predicts the Q ...
1
vote
1answer
46 views

Is it possible to prove that the target policy is better than the behavioural policy based on learned Q values?

I have retrospective data for a sort of "behaviour policy" which I will use to train a deep q network to learn a target greedy policy. After learning the Q values for this target policy, can we make ...
0
votes
2answers
94 views

When we use a neural network to approximate the Q values, is the Q target a single value?

I have two questions When we use our network to approximate our Q values, is the Q target a single value? During backpropagation, when the weights are updated, does it automatically update the Q ...
0
votes
0answers
27 views

My Double DQN with Experience Replay produces a no-action decision most of the time. Why?

I've written a Double DQN-based stock trading bot using mainly time series stock data. The internal network of the Double DQN is a LSTM which handles the time series data. An Experience Replay buffer ...