Questions tagged [dqn]

For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

how to calculate the virtual time of a reinforcement learning model

I have heard about some RL models like alpha go being trained for days but in reality has gained thousands of years of experience and a model which teaches a 3d figure how to walk and fight being ...
user avatar
-1 votes
0 answers
33 views

Why do we use mean squared error loss with deep q networks?

For computing the TD error (pseudo) loss in DQNs, we have the following formula - $$L(\theta) = 0.5*(R_{t+1} + \gamma[max_aq_{\theta}(S_{t+1}, a)] - q_{\theta}(S_t, A_t))^2$$ However, in practice, we ...
user avatar
0 votes
0 answers
19 views

Double DQN performs worse than DQN

I have an agent that has to explore a customized environment. The environment is a grid (100 squares horizontally, 100 squares vertically, each square is 10 meters wide). In the environment, there are ...
user avatar
  • 146
0 votes
1 answer
48 views

Which paper describes the effect of learning_starts in Reinforcement Learning?

I have seen many popular RL libraries have a learning_start parameter. This allows the agent to collect enough experiences before training on the replay_buffer. However, I am unable to find the paper ...
user avatar
0 votes
0 answers
36 views

What does maximal |Q| mean in DQN?

I am reading this paper and came across the term maximal |Q|. I'd like to know whether it refers to the Q values of the current state $Q(s_t,a_t)$ or that of the target $\mathbb{max}_aQ(S_{t+1}, A_t)$....
user avatar
0 votes
1 answer
50 views

Has there been a study done in tuning hyper-parameters for off-policy reinforcement learning?

I am interested in learning about hyper-parameter tuning for off-policy reinforcement learning (specifically DQN). Could someone point me to papers published or empirical observations in this area?
user avatar
1 vote
1 answer
47 views

Time taken to solve cartpole environment using DQN

I am trying to solve the cartpole environment (GitHub) using DQN agent. I have been building my own DQN agent by following a tutorial by Jon Krohn. I am able to solve the environment with a maximum ...
user avatar
1 vote
0 answers
19 views

What Kind of Reinforcement Learning Algorithms Can Be Used when the Action Space is Unfeasibly Large?

I know Deep Q network as a $S\times A$ DNN which maps the $S$ dimensional statespace to q-values of $A$ distinct actions. In my problem, the action space is still discrete, and finite, but depending ...
user avatar
  • 111
3 votes
1 answer
176 views

Why do Q-values diverge without a target network?

After reviewing similar posts on this topic, I understand that a target network is used to prevent "divergence", but am not sure what it actually means. Q-values are predicted using a ...
user avatar
2 votes
0 answers
62 views

How does one detect training instabilities in DQN?

I am curious what training instabilities look like in a standard dqn, with or without a target network. I'm assuming the loss function would never converge since the difference between the predicted q-...
user avatar
0 votes
0 answers
27 views

How to quickly train your agent to get a sense of the problem?

I have a custom implementation of DQN. My robot/agent is running in Gazebo with ROS. Though I am trying a very simple task of pushing a cube, the DQN agent is taking too much time. DQN of 300 episode ...
user avatar
-1 votes
1 answer
25 views

How to form 10 and 20 actions in corridor environment, in the paper "Dueling Network Architectures for Deep Reinforcement Learning"? [closed]

I am just reading a paper titled "Dueling Network Architectures for Deep Reinforcement Learning". In this paper, 4. Experiment (4.1 Policy evaluation), I just wonder how to form 5, 10, and ...
user avatar
1 vote
0 answers
25 views

Reinforcement Learning with constant reward in constant episodes time length

I have a situation where I'm trying to maximize the number of steps in a fixed training time frame. It's possible that specific steps will lead to a delay until the agent can act again, and thus less ...
user avatar
1 vote
1 answer
53 views

Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?

The creation of negative rewards leads to the chance of Q-values being negative. However, networks with relu or sigmoid activations, just cannot predict negative values. This will lead to a case where ...
user avatar
0 votes
1 answer
41 views

How to deal with changing rewards in Q-learning? DQN?

I read the working of Q-learning through a grid-based taxi routing wherein a taxi has to pick and drop off a passenger from source to destination. Likewise, I have a routing problem and hence, I tried ...
user avatar
2 votes
1 answer
134 views

Why do we use "true labels" that are based on the output of our network in Deep Q-Learning?

In the original DQN paper, the $\ell_2$ loss is taken over the distance between our network output, $\hat{q}(s_j,a_j,w)$ and the labels $y_j=r_j+\gamma \cdot \max\limits_{a'} \hat{q}(s_{j+1},a',w^-)$, ...
user avatar
0 votes
0 answers
22 views

DDQN for Connect 4: Sudden explosion of Loss

I am trying to solve Connect 4 with DDQN through the self-play regime that was used for AlphaZero. That means, I let a student version play against a teacher version of itself and replace the teacher ...
user avatar
  • 31
3 votes
1 answer
56 views

What is the total number of actions and rewards count

Reading this two articles about Reinforcement Learning: Deep Reinforcement Learning with Double Q-learning by Hado van Hasselt et al. Human-level control through deep reinforcement learning by ...
user avatar
  • 133
0 votes
0 answers
27 views

When Hindsight Experience Replay is deployed, does the input need to be augmented as well?

In Hindsight Experience Replay (HER), we augment the state representation $s_t$ with some goal $g_T$, which corresponds to the state reached after $T$ steps, such that $s'_t = s_t || g_T$. Later, some ...
user avatar
  • 81
1 vote
0 answers
97 views

What is commonly done for standardization/normalization of the targets in Deep Q-Learning?

I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
user avatar
  • 33
0 votes
1 answer
160 views

When calculating the max in DQN, do I have to calculate the Q for every possible action for a particular state?

I'm trying to implement the DQN paper using python/pytorch for my needs (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). I'm studying the main algorithm: I am a bit confused about the $\gamma* \max ...
user avatar
3 votes
1 answer
85 views

How do I design the network for Deep Q-Network?

I am playing with deep q-learning and I am thinking about what the proper architecture of a network used for deep q-learning is. I have a very simple environment, basically a 18x18 matrix, where 3 ...
user avatar
  • 133
1 vote
1 answer
77 views

Doesn't the n-step Tree Backup algorithm negatively affect the DQN-Agent by creating inconsistent look-ahead targets?

In the text book of Sutton and Barto on page 152 they introduce the n-step Tree Backup algorithm, where the tree-backup n-step return is defined via $$ G_{t:t+n} = R_{t+1} + \gamma \sum_{a \neq A_{t+1}...
user avatar
  • 33
1 vote
0 answers
39 views

Is there a way use DQN to find the optimal combination of actions (control inputs) and environment parameters?

I am using DQN to find the optimal sequence of control inputs to a dynamic system. The setup is as follows: At the beginning of each episode, the system is initialized to the SAME initial condition ...
user avatar
1 vote
1 answer
43 views

Can directly using expert policy in epsilon-greedy speed-up Q-learning?

In deep Q-learning we typically use epsilon-greedy policy during training. We choose a random action for a certain probability $\epsilon$, and choose the action that maximize the current Q-value ...
user avatar
  • 183
1 vote
1 answer
39 views

Is using Monte-Carlo estimate of returns in Deep Q Learning possible?

In all the tutorials of deep Q-learning (using neural networks) I have read so far, the state-action value function $Q(s,a)$ is learned by temporal difference learning. However, in policy gradient ...
user avatar
  • 183
2 votes
1 answer
81 views

Deep Q-Learning Model Effectiveness Improves then Crashes

I am implementing a Deep Q-Learning Algorithm. The model appears to improve but after awhile it just crashes and does just as well as if an agent was making random decisions. Shouldn't the behavior ...
user avatar
3 votes
2 answers
243 views

How to deal with changing environment in reinforcement learning

I am new to RL and I'm currently working on implementing a DQN and DDPG agent for a 2D car parking environment. I want to train my agent so that it can successfully traverse the env and park in the ...
user avatar
0 votes
1 answer
53 views

How to define actions on a list of values?

For a DQN algorithm, where my state is a list of values, say: [5, 3, 4, 7, 8, 2, 6] How can I define an action space that allows me to move a value in the list from one position to another? For ...
user avatar
  • 3
0 votes
1 answer
41 views

Deep RL reward design for neuron centerline extraction task

As part of a bigger scope project, I'm training a RL agent that attempts to reconstruct, pixel by pixel, the trajectory of a neuron on a segmented image. To give a better insight on the task that I'm ...
user avatar
1 vote
1 answer
275 views

Using "softmax" (non-linear) vs "linear" activation function in Deep Reinforcement Learning

I am following the tutorial in this video: https://youtu.be/cO5g5qLrLSo which implements deep reinforcement learning (DQN) to balance cart pole in OpenAI default environment. The DQN model looks like ...
user avatar
0 votes
1 answer
149 views

Action selection in Batch-Constrained Deep Q-learning (BCQ)

For simplicity, let's consider the discrete version of BCQ where the paper and the code are available. In the line 5 of Algorithm 1 we have the following: $$ a' = \text{argmax}_{a'|G_{\omega}(a', s')/\...
user avatar
  • 81
1 vote
1 answer
49 views

How Come My (D)DQN Fails To Learn?

I am currently trying to teach a (D)DQN algorithm to play a 10x10 GridWorld game, so I can compare the two as I increase the number of moves the agent can take. The rewards are as follows: A step = -1 ...
user avatar
1 vote
1 answer
92 views

How to output an integer/discrete number n with a single output neuron?

Say I have a game with 4 base actions [left, right, up, down] and then a value n, which determines how many times the chosen action is repeated. For example, action = left, n = 3 -> go left 3 times....
user avatar
0 votes
1 answer
40 views

Does "number of actions" refer to the number of actions taken or size of the action space?

In the original DDQN article (https://arxiv.org/pdf/1509.06461.pdf,) the phrase "number of actions" is used twice; First, in the following context: Secondly in Theorem 1. I have a hard ...
user avatar
0 votes
0 answers
38 views

How Does The DDQN Step/interact In The Environment?

I have made a (D)DQN Model. In this model, regardless of whether I initialize it in DDQN or DQN mode, it uses an experience replay memory. The way I gather transitions for this experience replay ...
user avatar
1 vote
1 answer
51 views

Would it be possible to enforce the same $s_{t + 1}$ between the model's estimate and the target function's Q-value?

Say I have a game of blackjack, and I am trying to teach a single forward-pass neural network to approximate the Q value of the current state and action. There are 3 inputs: The current card in hand, ...
user avatar
1 vote
0 answers
60 views

Reinforcement Learning applied to Optimisation Problem

Problem Statement: We are given an optimisation problem; with production centres, source airport, destination airports, transfer points and finally delivered to the customers. This is better explained ...
user avatar
  • 11
0 votes
0 answers
70 views

Should you clip Q values if they start to grow indefinitely?

I am training the SAC algorithm for an environment where the rewards are small as shown below and the episode length is 84. I have a problem with the Q values that grow with each step. The following ...
user avatar
  • 197
0 votes
0 answers
32 views

dDQN converges, but makes suboptimal decisions that are mainly focused on retrieving short-term reward

I use a dDQN to dispatch drivers in a ride-hailing environment. The action space has size (#drivers + 1), which means we can choose one of the drivers or choose to refuse an order (and wait for the ...
user avatar
  • 3
4 votes
1 answer
630 views

Do we use validation and test sets for training a reinforcement learning agent?

I am pretty new to reinforcement learning and was working with some code for the PPO and DQN algorithms. After looking at the code, I noticed that the authors did not include any code to setup a ...
user avatar
  • 187
1 vote
0 answers
67 views

Can Q-learning be used for my scenario, and how might I do so?

I have already asked 2-3 general questions w.r.t Q learning and now I am asking a scenario specific one. I will try to be concise and understandable. I really really need help. Scenario: I have a ...
user avatar
0 votes
0 answers
47 views

Alternatives to neural networks for function approximation in Q learning?

I want to know if there is anything other than neural networks (or Deep NNs) that I can effectively use to perform function approximation? I am asking this w.r.t to the use of approximators in Q ...
user avatar
2 votes
0 answers
45 views

How rewards are playing role in Deep Q Network

I have started working on Reinforcement Learning, specifically DQN. And I have watched some interesting videos on it. However, I have some doubts about how the model works. Let's say we are playing ...
user avatar
-1 votes
1 answer
66 views

What does the line of code "self.buffer[-1] = observation" do in this BufferWrapper class for DQN?

So the code is related to using a buffer ...
user avatar
5 votes
0 answers
722 views

Why isn't a target network used for the critic in on-policy actor-critic methods?

Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow: $$ V_\pi(s_t)...
user avatar
1 vote
1 answer
140 views

How can I use a ResNet as a function approximator for pixel based reinforcement learning?

I'd like to use a residual network to improve learning in image-based reinforcement learning, specifically on Atari Games. My main question is divided into 3 parts. Would it be wise to integrate a ...
user avatar
1 vote
0 answers
23 views

Transferring a Q-learning policy to larger instances

How do I best transfer and fine-tune a Q-learning policy that was trained on small instances to large instances? Some more details on the problem: I am currently trying to derive a decision policy for ...
user avatar
0 votes
1 answer
86 views

Are the Q-values of DQN bounded at a single timestep?

Consider that we have an agent that has a set of thousands of different actions at each timestep. The reward function in $R:S \rightarrow\{0,1\}$. Let $Q_{t}^\pi(s,a)$ be the estimate from the neural ...
user avatar
  • 889
1 vote
1 answer
84 views

Implementing Multiple NNs in one DQN model?

I'm trying to build a DQN Agent to take a set of 10 best actions simultaneously (integer values from 1 to 100) as outputs per episode. The input is a float. The goal is to find the optimal combination ...
user avatar
  • 13

1
2 3 4 5 6