Questions tagged [dqn]

For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.

Filter by
Sorted by
Tagged with
0
votes
1answer
79 views

Handle non-existing states in q-learning

I am using Q-learning to solve an engineering problem. The objective is to generate a Q-table associating states to Q-values. I created a State vector ...
3
votes
1answer
43 views

How would researchers determine the best deep learning model if every run of the code yields different results?

There are many factors that cause the results of ML models to be different for every run of the same piece of code. One factor could be different initialization of weights in the neural network. ...
1
vote
1answer
57 views

Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?

I ran a deep q learning algorithm (DQN) for $x$ number of epochs and got policy $\pi_1$. I reran the same script for the same $x$ number of epochs and got policy $\pi_2$. I expected $\pi_1 $ and $\...
3
votes
1answer
77 views

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

Twin Delayed Deep Deterministic (TD3) policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained ...
3
votes
1answer
30 views

If agent chooses an action that the environment can't operate, how should I handle this situation?

I'm building a really simple experiment, letting an agent move from the bottom-left corner to the upper-right corner on a 3x3 squared paper. I plan to use DQN to do this. I'm having trouble handling ...
1
vote
0answers
17 views

Does the concept of validation loss apply to training deep Q networks?

In deep learning, the concept of validation loss is to ensure that the model being trained is not currently overfitting the data. Is there a similar concept of overfitting in deep q learning? Given ...
0
votes
0answers
16 views

DQN not showing the agent is learning in a snake grid environment game

I've been trying to train a snake for the snake game in DQN. Which the snake can essentially just move up, down, left and right. I'm having a hard time getting the snake to stay alive longer. So my ...
2
votes
1answer
42 views

How to evaluate a Deep Q-Network

Good day, it's a pleasure having joined this Stack. In my master thesis I have to expand a Deep Reinforcement Learning Network, to be precise a Deep Q-Network, which is used to control machines in an ...
2
votes
2answers
150 views

What should the target be when the neural network outputs multiple Q values in deep Q-learning?

I have some gaps in my understanding regarding the performing of the gradient descent in Deep - Q networks. The original deep q network for Atari performs a gradient descent step to minimise $y_j - Q(...
1
vote
1answer
28 views

Is it possible to prove that the target policy is better than the behavioural policy based on learned Q values?

I have retrospective data for a sort of "behaviour policy" which I will use to train a deep q network to learn a target greedy policy. After learning the Q values for this target policy, can we make ...
1
vote
1answer
46 views

Why can't DQN be used for self-driving cars?

Why can't DQN be used for self-driving cars? Why can't DQN and similar RL algorithms be used for self-driving cars? The reason why I am curious is that it successfully plays go and other multistate ...
1
vote
1answer
22 views

Do smaller loss values during DQN training produce better policies?

During the training of DQN, I noticed that the model with prioritized experience replay (PER) had a smaller loss in general compared to a DQN without PER. The mean squared loss was an order of ...
0
votes
0answers
10 views

How is the parameterised server updated in distributed DQN?

In this paper about Massively Parallel Methods for Deep Reinforcement Learning, the parallelisation of DQN is done via separating the actors and learners. Multiple actors carry out the $\epsilon$ ...
1
vote
1answer
42 views

How does the DQN loss from td_targets against q_values make sense?

Why td_loss is calculated from (td_targets against q_values)? Why I am lost is because: <...
0
votes
0answers
25 views

Reinforcement learning random agent always performing the same few actions

I have a DQN model which has 3 features as inputs (properly normalized) and should output q-values for each of the 100 possible actions. However, prior to any training, I would like to examine the ...
4
votes
1answer
48 views

How to represent players in a multi agent environment so each model can distinguish its own player

So I have 2 models trained with DQN algorithm that I want to train in a multi-agent environment to see how they react with each other. The models were trained in an environment consisting of 0's and 1'...
1
vote
0answers
22 views

How can I design a DQN or policy gradient model to explore and collect all optimal solutions?

I am working to use DQN and Policy Gradient reinforcement learning models to solve classic maze escaping problems. So far, I have been able to train a model, which, after around 100 episodes, ...
1
vote
0answers
143 views

Replace epsilon greedy action selection and the standard DQN by an Independent Gaussian Noise Network Model

Here is my code Recently, I solved the game of Atari Breakout using a classic DQN model. The convergence of the mean reward slowly improved during three days. I was interested in learning a method ...
1
vote
0answers
17 views

Why are Dueling Q Networks not used more often to approximate Q-values in reinforcement learning algorithms?

I've just learned about Dueling Network Architectures to estimate $Q$-values and am wondering why this architecture is not used more often in deep RL algorithms? DDPG and TD3 estimate the $Q$-function ...
4
votes
3answers
245 views

What is the target Q-value in DQNs?

I understand that in DQNs, the loss is measured by taking the MSE of outputted Q-values and target Q-values. Whats does the target Q-values represent? And how is it obtained/calculated by the DQN?
3
votes
1answer
53 views

Are Q values estimated from a DQN different from a duelling DQN with the same number of layers and filters?

I am confused about the Q values of a duelling deep Q network (DQN). As far as I know, duelling DQNs have 2 outputs Advantage: how good it is to be in a particular state $s$ Value: the advantage of ...
2
votes
1answer
35 views

How does the optimization process in hindsight experience replay exactly work?

I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the ...
1
vote
0answers
31 views

Why isn't my DQN agent improving when trained on Atari Breakout?

Lately, I have implemented DQN for Atari Breakout. Here is the code: https://github.com/JeremieGauthier/AI_Exercices/blob/master/Atari_Breakout/DQN_Breakout.py I have trained the agent for over ...
2
votes
0answers
28 views

How was the DQN trained to play many games?

Some people claim that DQN was used to play many Atari games. But what actually happened? Was DQN trained only once (with some data from all games) or was it trained separately for each game? What was ...
0
votes
0answers
14 views

DQN is unable to learn from image data

I am trying to write a DQN model that will be able to solve OpenAI gym CartPole environment. I successfully managed to do it using scalar observation data that ...
0
votes
0answers
25 views

How should I define the loss function when using DQN to estimate the probability density?

I'm doing a Deep Q-learning project. All of my rewards are positive and there are two terminal states. One of them has a zero reward and the other has a high positive reward. The rewards are ...
1
vote
1answer
36 views

Can this be a possible deep q learning pseudocode?

I am not using replay here. Can this be a possible deep q learning pseudocode? ...
1
vote
0answers
29 views

Atari Breakout Infrastructure

This is how they describe their infrastructure in https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf. I want to implement the game of Atari Breakout. ...
2
votes
0answers
36 views

How much time does it take to train DQN on Atari environment?

I am trying to build a DQN model for the Atari Pong game, but I am not sure whether the model is learning at all. I am using the architecture described in the paper Playing Atari with Deep ...
1
vote
0answers
11 views

Whats the correct loss function to use during deep Q-learning (discrete action space)

After playing around with normal Q-learning I have decided to switch to deep Q-learning and I have encountered this problem. As I understand, for a task with discrete action space, where there are 4 ...
3
votes
2answers
100 views

What are some online courses for deep reinforcement learning?

What are some (good) online courses for deep reinforcement learning? I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
2
votes
0answers
29 views

Why is my DQN model not getting better?

I've created a deep Q network. My model does not get better, and can't see what I'm doing wrong. I'm new to RL. Replay Memory ...
3
votes
0answers
31 views

Representation of state space, action space and reward system for Reinforcement Learning problem

I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without ...
1
vote
1answer
78 views

Why isn't my implementation of DQN using TensorFlow on the FrozenWorld environment working?

I am trying to test DQN on FrozenWorld environment in gym using TensorFlow 2.x. The update rule is (off policy) $$Q(s,a) \leftarrow Q(s,a)+\alpha (r+\gamma~ max_{a'}Q(s',a')-Q(s,a))$$ I am using an ...
2
votes
1answer
153 views

Deep Reinforcement Learning: Rewards suddenly dip down

I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using ...
2
votes
0answers
53 views

How to correctly implement self-play with DQN?

I have an environment where an agent faces an equal opponent, and while I've achieved OK performance implementing DQN and treating the opponent as a part of the environment, I think performance would ...
3
votes
1answer
40 views

Can experience replay be used for training after completing every single epoch?

The DQN implements replay memory. Based on my research, I believe the replay memory starts to get used for training once there is enough experience in the memory buffer. This means the neural network ...
2
votes
1answer
35 views

Intutitive explanation of why Experience Replay is used in a Deep Q Network?

I understand that Experience Replay is used for data efficiency reasons and to remove correlations in sequences of data. How exactly do these sequences of correlated data affect the performance of the ...
2
votes
1answer
67 views

How is the expected value in the loss function of DQN approximated?

In Deep Q Learning the parametrized Q-functions $Q_i$ are optimised by performing gradient descent on the series of loss functions $L_i(\theta_i)= E_{(s,a)\sim p}[(y_i-Q(s,a;\theta_i))^2]$ , where ...
1
vote
0answers
119 views

What's the difference in using multiple convolutional layers and no pooling versus using a single convolutional layer and a single max pooling layer?

I'm currently working on a college project in which I'm designing a Deep Q-Network that takes images/frames as an input. I've been searching online to see how other people have designed their ...
1
vote
1answer
100 views

Understanding the loss function in deep Q-learning

I am trying to understand how deep Q learning (DQN) works. To my current understanding, each $Q(s, a)$ functions is estimated to be a function of a feature vector of its state $\phi$(s) and the weight ...
2
votes
1answer
52 views

NoisyNet DQN with default parameters not exploring

I implemented a DQN algorithm that plays OpenAIs Cartpole environment. The NN architecture consists of 3 normal linear layers that encode the state, and one noisy linear layer, that predicts the Q ...
5
votes
0answers
70 views

Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?

While reading the DQN paper, I found that randomly selecting and learning samples reduced divergence in RL using a non-linear function approximator (e.g a neural network). So why does Reinforcement ...
4
votes
1answer
49 views

How can a DQN backpropagate its loss?

I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I ...
2
votes
4answers
158 views

How to stop DQN Q function from increasing during learning?

Following the DQN algorithm with experience replay: We calculate the $loss=(Q(s,a)-(r+Q(s+1,a)))^2$. Assume I have positive but changing rewards. Meaning, $r>0$. Thus, since the rewards are ...
1
vote
0answers
32 views

Is the training of multi-version of the same system at the same time affecting the results?

I'm using DQN to train multi-version of the same system and there is a small difference when I run them both separately. However, my result suddenly dropped in both versions if I run them both at the ...
1
vote
1answer
79 views

DQN card game - how to represent the actions?

I want to train a DQN card game named witches. It consists of 60 Cards (1-14 of Yellow, Blue, Green, Red Cards + 4 Wizzards). The color of the first layed card has to be respected by the other players ...
3
votes
0answers
45 views

DQN, how to choose the reward fucntion?

I built a simple AI system that tries to solve the 8 puzzle using DQN. The problem is, if the agent gets only a reward greater than zero when winning, the training will take a long time, so I made a ...
1
vote
0answers
67 views

Why isn't this deep Q network agent for the snake game learning?

I wanted to combine this snake game with this DQN implementation I found in this article. First, I tried to change the NN's input layer to a 400 input. The game has a field of 20 times 20, so I ...
2
votes
0answers
26 views

What could be the cause of the drop of the total reward when using DQN to solve the cart-pole environment?

I'm trying to use DQN to solve the cart-pole environment. I have 2 networks (target and behavior). Both of them have 3 hidden layers with 24 neurons, using the ReLU activation. The loss is MSE and the ...