Questions tagged [dqn]

For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.

Filter by
Sorted by
Tagged with
3 votes
1 answer
40 views

What is the total number of actions and rewards count

Reading this two articles about Reinforcement Learning: Deep Reinforcement Learning with Double Q-learning by Hado van Hasselt et al. Human-level control through deep reinforcement learning by ...
user avatar
  • 133
0 votes
0 answers
19 views

When Hindsight Experience Replay is deployed, does the input need to be augmented as well?

In Hindsight Experience Replay (HER), we augment the state representation $s_t$ with some goal $g_T$, which corresponds to the state reached after $T$ steps, such that $s'_t = s_t || g_T$. Later, some ...
user avatar
  • 71
1 vote
0 answers
39 views

What is commonly done for standardization/normalization of the targets in Deep Q-Learning?

I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
user avatar
  • 23
0 votes
1 answer
49 views

When calculating the max in DQN, do I have to calculate the Q for every possible action for a particular state?

I'm trying to implement the DQN paper using python/pytorch for my needs (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). I'm studying the main algorithm: I am a bit confused about the $\gamma* \max ...
user avatar
3 votes
1 answer
60 views

How do I design the network for Deep Q-Network?

I am playing with deep q-learning and I am thinking about what the proper architecture of a network used for deep q-learning is. I have a very simple environment, basically a 18x18 matrix, where 3 ...
user avatar
  • 133
1 vote
1 answer
50 views

Doesn't the n-step Tree Backup algorithm negatively affect the DQN-Agent by creating inconsistent look-ahead targets?

In the text book of Sutton and Barto on page 152 they introduce the n-step Tree Backup algorithm, where the tree-backup n-step return is defined via $$ G_{t:t+n} = R_{t+1} + \gamma \sum_{a \neq A_{t+1}...
user avatar
  • 23
1 vote
0 answers
20 views

Is there a way use DQN to find the optimal combination of actions (control inputs) and environment parameters?

I am using DQN to find the optimal sequence of control inputs to a dynamic system. The setup is as follows: At the beginning of each episode, the system is initialized to the SAME initial condition ...
user avatar
1 vote
1 answer
32 views

Can directly using expert policy in epsilon-greedy speed-up Q-learning?

In deep Q-learning we typically use epsilon-greedy policy during training. We choose a random action for a certain probability $\epsilon$, and choose the action that maximize the current Q-value ...
user avatar
  • 173
1 vote
1 answer
30 views

Is using Monte-Carlo estimate of returns in Deep Q Learning possible?

In all the tutorials of deep Q-learning (using neural networks) I have read so far, the state-action value function $Q(s,a)$ is learned by temporal difference learning. However, in policy gradient ...
user avatar
  • 173
2 votes
1 answer
71 views

Deep Q-Learning Model Effectiveness Improves then Crashes

I am implementing a Deep Q-Learning Algorithm. The model appears to improve but after awhile it just crashes and does just as well as if an agent was making random decisions. Shouldn't the behavior ...
user avatar
2 votes
2 answers
96 views

How to deal with changing environment in reinforcement learning

I am new to RL and I'm currently working on implementing a DQN and DDPG agent for a 2D car parking environment. I want to train my agent so that it can successfully traverse the env and park in the ...
user avatar
0 votes
1 answer
44 views

How to define actions on a list of values?

For a DQN algorithm, where my state is a list of values, say: [5, 3, 4, 7, 8, 2, 6] How can I define an action space that allows me to move a value in the list from one position to another? For ...
user avatar
  • 3
0 votes
1 answer
40 views

Deep RL reward design for neuron centerline extraction task

As part of a bigger scope project, I'm training a RL agent that attempts to reconstruct, pixel by pixel, the trajectory of a neuron on a segmented image. To give a better insight on the task that I'm ...
user avatar
1 vote
1 answer
109 views

Using "softmax" (non-linear) vs "linear" activation function in Deep Reinforcement Learning

I am following the tutorial in this video: https://youtu.be/cO5g5qLrLSo which implements deep reinforcement learning (DQN) to balance cart pole in OpenAI default environment. The DQN model looks like ...
user avatar
0 votes
1 answer
101 views

Action selection in Batch-Constrained Deep Q-learning (BCQ)

For simplicity, let's consider the discrete version of BCQ where the paper and the code are available. In the line 5 of Algorithm 1 we have the following: $$ a' = \text{argmax}_{a'|G_{\omega}(a', s')/\...
user avatar
  • 71
0 votes
1 answer
33 views

How Come My (D)DQN Fails To Learn?

I am currently trying to teach a (D)DQN algorithm to play a 10x10 GridWorld game, so I can compare the two as I increase the number of moves the agent can take. The rewards are as follows: A step = -1 ...
user avatar
1 vote
1 answer
39 views

How to output an integer/discrete number n with a single output neuron?

Say I have a game with 4 base actions [left, right, up, down] and then a value n, which determines how many times the chosen action is repeated. For example, action = left, n = 3 -> go left 3 times....
user avatar
0 votes
1 answer
32 views

Does "number of actions" refer to the number of actions taken or size of the action space?

In the original DDQN article (https://arxiv.org/pdf/1509.06461.pdf,) the phrase "number of actions" is used twice; First, in the following context: Secondly in Theorem 1. I have a hard ...
user avatar
0 votes
0 answers
34 views

How Does The DDQN Step/interact In The Environment?

I have made a (D)DQN Model. In this model, regardless of whether I initialize it in DDQN or DQN mode, it uses an experience replay memory. The way I gather transitions for this experience replay ...
user avatar
1 vote
1 answer
50 views

Would it be possible to enforce the same $s_{t + 1}$ between the model's estimate and the target function's Q-value?

Say I have a game of blackjack, and I am trying to teach a single forward-pass neural network to approximate the Q value of the current state and action. There are 3 inputs: The current card in hand, ...
user avatar
1 vote
0 answers
52 views

Reinforcement Learning applied to Optimisation Problem

Problem Statement: We are given an optimisation problem; with production centres, source airport, destination airports, transfer points and finally delivered to the customers. This is better explained ...
user avatar
  • 11
0 votes
0 answers
39 views

Should you clip Q values if they start to grow indefinitely?

I am training the SAC algorithm for an environment where the rewards are small as shown below and the episode length is 84. I have a problem with the Q values that grow with each step. The following ...
user avatar
  • 167
0 votes
0 answers
28 views

dDQN converges, but makes suboptimal decisions that are mainly focused on retrieving short-term reward

I use a dDQN to dispatch drivers in a ride-hailing environment. The action space has size (#drivers + 1), which means we can choose one of the drivers or choose to refuse an order (and wait for the ...
user avatar
  • 3
4 votes
1 answer
353 views

Do we use validation and test sets for training a reinforcement learning agent?

I am pretty new to reinforcement learning and was working with some code for the PPO and DQN algorithms. After looking at the code, I noticed that the authors did not include any code to setup a ...
user avatar
  • 187
1 vote
0 answers
59 views

Can Q-learning be used for my scenario, and how might I do so?

I have already asked 2-3 general questions w.r.t Q learning and now I am asking a scenario specific one. I will try to be concise and understandable. I really really need help. Scenario: I have a ...
user avatar
0 votes
0 answers
33 views

Alternatives to neural networks for function approximation in Q learning?

I want to know if there is anything other than neural networks (or Deep NNs) that I can effectively use to perform function approximation? I am asking this w.r.t to the use of approximators in Q ...
user avatar
2 votes
0 answers
40 views

How rewards are playing role in Deep Q Network

I have started working on Reinforcement Learning, specifically DQN. And I have watched some interesting videos on it. However, I have some doubts about how the model works. Let's say we are playing ...
user avatar
-1 votes
1 answer
54 views

What does the line of code "self.buffer[-1] = observation" do in this BufferWrapper class for DQN?

So the code is related to using a buffer ...
user avatar
4 votes
0 answers
679 views

Why isn't a target network used for the critic in on-policy actor-critic methods?

Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow: $$ V_\pi(s_t)...
user avatar
1 vote
1 answer
91 views

How can I use a ResNet as a function approximator for pixel based reinforcement learning?

I'd like to use a residual network to improve learning in image-based reinforcement learning, specifically on Atari Games. My main question is divided into 3 parts. Would it be wise to integrate a ...
user avatar
1 vote
0 answers
21 views

Transferring a Q-learning policy to larger instances

How do I best transfer and fine-tune a Q-learning policy that was trained on small instances to large instances? Some more details on the problem: I am currently trying to derive a decision policy for ...
user avatar
0 votes
1 answer
67 views

Are the Q-values of DQN bounded at a single timestep?

Consider that we have an agent that has a set of thousands of different actions at each timestep. The reward function in $R:S \rightarrow\{0,1\}$. Let $Q_{t}^\pi(s,a)$ be the estimate from the neural ...
user avatar
  • 879
1 vote
1 answer
74 views

Implementing Multiple NNs in one DQN model?

I'm trying to build a DQN Agent to take a set of 10 best actions simultaneously (integer values from 1 to 100) as outputs per episode. The input is a float. The goal is to find the optimal combination ...
user avatar
  • 13
0 votes
1 answer
73 views

How to incorporate action information in the state input of a DQN?

I am working on an RL problem that I am trying to solve using a Deep Q-network. The problem concerns choosing drivers to take specific taxi orders. I am familiar with most of the existing works and ...
user avatar
  • 3
0 votes
1 answer
437 views

Deep Q-Learning with multiple discrete actions

I am working on a DQN project with Pytorch, where I should choose multiple discrete actions, each in a range, say, (0, 15). I am wondering how I can model it, such ...
user avatar
  • 1
2 votes
1 answer
384 views

Creating DQN Learning Agent without Gym environment for a custom project

In a project for college I created a simple turn based game, with up to 4 players that can either move or attack the opponents. The players are playing over the network, meaning the clients are ...
user avatar
0 votes
0 answers
42 views

Why don't I get the same results of Q-Learning as in Aurélion Géron's Hands-on Machine Learning book?

I noticed something rather intriguing while testing the Deep Q-Network implementation from Aurélion Géron's book Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition; I copy-...
user avatar
  • 489
0 votes
1 answer
46 views

Where can I find the original conference paper that introduced Q-learning and Deep Q-Learning?

I tried searching a lot, but I could neither find the paper that introduced Q-Learning nor the paper that introduced Deep Q Learning. If anyone knows anything about it please do tell me.
user avatar
1 vote
1 answer
69 views

Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?

I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it'...
user avatar
  • 489
0 votes
1 answer
25 views

In a DDQN architecture, why is the value of a state assumed to be the average of the Q values of the actions?

In a Dueling DQN agent (Wang et al.), the Q function is decomposed as $$ Q(s, a)=V(s) + A(s, a) - \frac{1}{|A|}\sum_{a'\in \mathcal{A}}A(s, a') $$ representing the value of the state, plus the ...
user avatar
  • 101
1 vote
1 answer
43 views

How to detect entities in Montezuma's Revenge environment

I'm thinking of implementing "Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation" paper. In this paper authors used some custom object ...
user avatar
  • 13
0 votes
0 answers
69 views

DQN learns to always choose the same action for all states

I have created an RL model that uses QBased policy with a neural network for estimating Q values. My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. ...
user avatar
0 votes
0 answers
40 views

CartPoleV0 model is not getting trained in even after 1500+ episodes using deep Q-learning

I am new to deep Q learning and trying to train the open AI cartpole_V0 game using deep Q learning. Here is my code: ...
user avatar
  • 101
4 votes
1 answer
997 views

Are policy gradient methods good for large discrete action spaces?

I have seen this question asked primarily in the context of continuous action spaces. I have a large action space (~2-4k discrete actions) for my custom environment that I cannot reduce down further: ...
user avatar
2 votes
0 answers
198 views

Update Rule with Deep Q-Learning (DQN) for 2-player games

I am wondering how to correctly implement the DQN algorithm for two-player games such as Tic Tac Toe and Connect 4. While my algorithm is mastering Tic Tac Toe relatively quickly, I cannot get great ...
user avatar
  • 31
0 votes
0 answers
119 views

Reward firstly increase, but after more episodes, start decrease, and weights diverges

I'm making a simple deep Q learning algorithm, with cartpole-v1 env. Like you can see in chart, after many episodes the reward decrease, some possible reasons? The exploration vs axplotation algorithm ...
user avatar
4 votes
1 answer
92 views

In DQN, would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?

In the classical examples of deep q-learning, I often see neural networks in which the input represents the state of the agent, while the output is a tuple with all the values of $Q(s, a)$ predicted ...
user avatar
-1 votes
1 answer
32 views

Using states (features) and actions from a heuristic model to estimate the value function of a reinforcement learning agent [closed]

new to RL here. As far as i understood from RL courses, that there is two sides of reinforcement learning. Policy Evaluation, which is the task of knowing the value function for certain policy. and ...
user avatar
  • 3
2 votes
1 answer
761 views

How to recover the target Q network's weights solely from the snapshots of the primary Q network's weights in DQN?

Suppose that I have a DQN agent, which has two neural networks: one is the primary Q network and the other is the target Q network. In every update, the target Q network is updated with a soft update ...
user avatar
  • 143
-2 votes
1 answer
273 views

Why the optimal Bellman operator of a Q-function can be approximated by a single point

I am currently studying reinforcement learning, especially DQN. In DQN, learning proceeds in such a way as to minimize the norm (least-squares, Huber, etc.) of the optimal Bellman equation and the ...
user avatar
  • 97

1
2 3 4 5 6