Questions tagged [dqn]

For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.

Filter by
Sorted by
Tagged with
2
votes
3answers
262 views

Reinforcement learning: How to deal with illegal actions? [duplicate]

I'm a beginner of RL and currently trying to make DQN agent that can act optimally in a simple situation. In the situation agent should decide at what rate to charge or discharge the electrical ...
2
votes
1answer
267 views

Deep Reinforcement Learning: Rewards suddenly dip down

I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using ...
2
votes
0answers
211 views

Gym dict space as keras DQN agent input

I'm trying to make an AI to play my own card game. I have an OpenAI gym for the game with Dict as an observation space. It is nested dict, so I can't easily replace it with a tuple. I want to pass ...
2
votes
0answers
100 views

Why experience reply memory in DQN instead of a RNN memory?

I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
1
vote
2answers
84 views

How does one know that a problem is “model-free” in reinforcement learning?

Consider this slide from a Stanford lecture on reinforcement learning. It states that a model is the agent's representation of how the world changes in response to the agent's action. I've been ...
1
vote
1answer
201 views

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

I have a problem which I believe can be described as a contextual bandit. More specifically, in each round, I observe a context from the environment consisting of five continuous features, and, ...
1
vote
1answer
108 views

How and when should we update the Q-target in deep Q-learning?

I have recently watched David silver's course, and started implementing the deep Q-learning algorithm. I thought I should make a switch between the Q-target and Q-current directly (meaning, every ...
1
vote
1answer
97 views

Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?

I ran a deep q learning algorithm (DQN) for $x$ number of epochs and got policy $\pi_1$. I reran the same script for the same $x$ number of epochs and got policy $\pi_2$. I expected $\pi_1 $ and $\...
1
vote
1answer
54 views

DQN regarding fitting off actions

While studying DQN, I rarely if ever see an explanation of what to fit off actions to. Meaning, if the highest valued action of 3 choices if the first, when training, what do we do with the other two ...
1
vote
2answers
176 views

Can gamma be greater than 1 in a DQN?

If I have a DQN, and I care A LOT about future rewards (moreso than current rewards), can I set gamma to a number greater than 1? Like 1.1 perhaps?
1
vote
1answer
46 views

What kind of problems is DQN algorithm good and bad for?

I know this is a general question, but I'm just looking for intuition. What are the characteristics of problems (in terms of state-space, action-space, environment, or anything else you can think of) ...
1
vote
2answers
72 views

Why not use the target network in DQN as the predictor after training

Target network in DQN is known to make the network more stable, and the loss is like "how good I'm now compared to using the target". What I don't understand is, if the target network is the ...
1
vote
1answer
55 views

What are the variables that need to be saved and loaded, so that a DQN model starts where it left off?

TensorFlow allows users to save the weights and the model architecture, however, that will be insufficient unless the values of certain other variables are also stored. For instance, in DQN, if $\...
1
vote
1answer
81 views

Why do we need target network in deep Q learning? [duplicate]

I already know deep RL, but to learn it deeply I want to know why do we need 2 networks in deep RL. What does the target network do? I now there is huge mathematics into this, but I want to know deep ...
1
vote
1answer
49 views

Why do we update the weights of the target network in deep Q learning?

I know we keep the target network constant during training to improve stability, but why exactly are we updating the weights of our target network? In particular, if we've already reached convergence, ...
1
vote
1answer
102 views

Handle non-existing states in q-learning

I am using Q-learning to solve an engineering problem. The objective is to generate a Q-table associating state to Q-values. I created a State vector ...
1
vote
1answer
38 views

Initialising DQN with weights from imitation learning rather than policy gradient network

In AlphaGo, the authors initialised a policy gradient network with weights trained from imitation learning. I believe this gives it a very good starting policy for the policy gradient network. the ...
1
vote
1answer
80 views

Reinforcement learning simple problem: agent not learning, wrong action

I am pretty new to RL and I am trying to code a simple RL task with pytorch. The goal/task is the following: The initial state is $t_o$ and the agent takes an action $\Delta_t$: $t_o +\Delta_t = t_1$. ...
1
vote
1answer
56 views

Why do my rewards reduce after extensive training using D3QN?

I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually ...
1
vote
1answer
41 views

What is the optimal exploration-exploitation trade-off in Q*bert?

I am training an RL agent with Deep Q-learning + Experience Replay on the Q*bert Atari environment. After 400,000 frames, my agent appears to have learned strategic information about the game, but ...
1
vote
1answer
24 views

How is weighted average computed in Deep Q networks

I was going through the Sutton book and they said the update formula for Q learning comes from the weighted average of the returns I.e New estimate= old estimate +alpha*[returns- old estimate] So by ...
1
vote
1answer
42 views

In DQN, when do the parameters in the Neural Network update based on the reward received?

I'm aware that we back-propagate after computing the loss between: The Neural Network Q values and the Target Network Q values However, all this is doing is updating the parameters of the Neural ...
1
vote
1answer
35 views

Why are Target Networks used in Deep Q-Learning as opposed to the Expected Value equation?

I understand we use a target network because it helps resolve issues regarding stability, however, that's not what I'm here to ask. What I would like to understand is why a target network is used as a ...
1
vote
1answer
62 views

Why does adding another network help in double DQN? [duplicate]

What is the idea behind double DQN? The target in double DQN is computed as follows $$ Y_{t}^{\text {DoubleQ }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1},...
1
vote
1answer
43 views

What is the role of embeddings in a deep recurrent Q network?

When describing the model architecture for a deep recurrent q network, the authors of the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning each agent consists of a recurrent ...
1
vote
1answer
120 views

DQN card game - how to represent the actions?

I want to train a DQN card game named witches. It consists of 60 Cards (1-14 of Yellow, Blue, Green, Red Cards + 4 Wizzards). The color of the first layed card has to be respected by the other players ...
1
vote
1answer
86 views

Deep Q Learning for Simple Game Not Effective

This is a follow-up question about one I asked earlier. The first question is here. Basically, I have a game where a paddle moves left and right to catch as much "food" as possible. Some food is good (...
1
vote
1answer
65 views

How to build a DQN agent which can be trained through interactive learning?

I am trying to create a chatbot whose dialogue policy model will be trained through reinforcement learning. Dialogue Policy is responsible for selecting the action to take based on the given state of ...
1
vote
1answer
443 views

DQN exploration strategy for large grid-world environment

My task involves a large grid-world type of environment (grid size may be $30\times30$, $50\times50$, $100\times100$, at the largest $200\times200$). Each element in this grid either contains a 0 or a ...
1
vote
1answer
558 views

DQN Breakout adding an extra negative reward to help training?

I'm trying to train a DQN, so I'm using OpenAI gym and Breakout (Breakout-v0). I have altered the reward supplied by the environment: If the episode is not completed fully, the agent gets a -10 ...
1
vote
1answer
173 views

Using a DQN with a variable amount of Valid Moves per turn for a Board Game

I have created a game on an 8x8 grid and there are 4 pieces which can move essentially like checkers pieces (Forward left or Forward right only). I have implemented a DQN in order to pull this off. ...
1
vote
0answers
22 views

How should I change the hyper-parameters of the C51 algorithm, in order to obtain higher reward?

I have a scenario where, in an ideal situation, the greedy approach is the best, but when non-idealities are introduced which can be learned, DQN starts doing better. So, after checking what DQN ...
1
vote
1answer
36 views

In reinforcement learning, is it possible to make some actions more likely?

In a general DQN framework, if I have an idea of some actions being better than some other actions, is it possible to make the agent select the better actions more often ?
1
vote
0answers
19 views

What adapts an algorithm to continuous or to discrete action spaces?

Some RL algorithms can only be used for environments with continuous action spaces (e.g TD3, SAC), while others only for discrete action spaces (DQN), and some for both REINFORCE and other policy ...
1
vote
1answer
39 views

Why do we minimise the loss between the target Q values and 'local' Q values?

I have a question regarding the loss function of target networks and current (online) networks. I understand the action value function. What I am unsure about is why we seek to minimise the loss ...
1
vote
0answers
26 views

Should the exploration rate be updated at the end of the episode or at every step?

My agent uses an $\epsilon$-greedy strategy to learn. The exploration rate (i.e. $\epsilon$) decays throughout the training. I've seen examples where people update $\epsilon$ every time an action is ...
1
vote
0answers
44 views

How does one stack multiple observations in the input layer of a convolutional neural network?

The paper, Deep Recurrent Q-Learning for Partially Observable MDPs, talks about stacking multiple observations in the input of a convolutional neural network. How does this exactly work? Do the ...
1
vote
0answers
20 views

Is it feasible to train a DQN with thousands of input ports?

I designed a DQN architecture for some problem. The problem has a parameter $m$ as the number of clients. In my situation, $m$ is large, $m\in\{100,200,\ldots,1000\}$. For this situation, the number ...
1
vote
0answers
45 views

Why scaling reward drastically affects performance?

I have devised an gridworld-like environment where a RL agent is tasked to cover all the blank squares by passing through them. Possible actions are up, down, left, right. The reward scheme is the ...
1
vote
1answer
58 views

How is exponential moving average computed in deep Q networks?

In normal Q-learning, the update rule is an implementation of the exponential moving average, which then converges to the optimal true Q values. However, looking at DQN, how exactly is the exponential ...
1
vote
0answers
32 views

How can deep Q-learning converge if the targets may not be correct?

In deep Q-learning, $Q(s, a)$ and $Q'(s, a)$ are predicted or estimated by the neural network itself. In supervised learning, the target value is a true unbiased value. However, this isn't the case in ...
1
vote
0answers
69 views

What happens if our target network overestimates the value?

When we use DDQN, we often use the target network in case our online network overestimates a value, but this doesn't make sense to me, because What happens if our target network is the one that ...
1
vote
0answers
55 views

Atari Games: Pretrained CNN to accelerate training?

DQN for Atari takes considerable training time. For example, the 2015 paper in Nature notes that algorithms are trained for 50 million frames or equivalently around 38 days of game experience in total....
1
vote
0answers
33 views

Is there a way to show convergence of DQN other than by eye observation?

I made a DQN model and plot its reward curve. You can see intuitively that the curve already converged since its reward value now just oscillates. How can I show confidence that my DQN already reached ...
1
vote
0answers
40 views

Designing a reward function for my reinforcement learning problem

I'm working on a project lately and I'm trying to solve a problem with reinforcement learning and I have serious issues with shaping the reward function. The problem is designing a device with maximum ...
1
vote
0answers
71 views

How can the target rely on untrained parameters?

I'm trying to understand DQN. I understand where the loss function comes from. I'm just unsure about why the target function works in practice. Given the loss function $$ L_i(\theta_i) = [(y_i - Q(s,a;...
1
vote
0answers
82 views

Understanding the role of the target network in this DQN algorithm

I've found online this interesting algorithm: From what I understand reading this algorithm, I can't figure out why I should "perform the opposite action" and consequently storing that second ...
1
vote
0answers
36 views

DQN not showing the agent is learning in a snake grid environment game

I've been trying to train a snake for the snake game in DQN. Which the snake can essentially just move up, down, left and right. I'm having a hard time getting the snake to stay alive longer. So my ...
1
vote
1answer
56 views

Is it possible to prove that the target policy is better than the behavioural policy based on learned Q values?

I have retrospective data for a sort of "behaviour policy" which I will use to train a deep q network to learn a target greedy policy. After learning the Q values for this target policy, can we make ...
1
vote
1answer
67 views

Do smaller loss values during DQN training produce better policies?

During the training of DQN, I noticed that the model with prioritized experience replay (PER) had a smaller loss in general compared to a DQN without PER. The mean squared loss was an order of ...