Questions tagged [dqn]

For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.

Filter by
Sorted by
Tagged with
1 vote
0 answers
12 views

Why slow-changing policy invalidates Double DQN approach in TD3 paper?

In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
Jerry Ding's user avatar
2 votes
0 answers
37 views

Why does only Deep Q Learning have an overestimation bias?

There is a lot of discussion about the overestimation bias for Deep Q Learning and similar off-policy action value estimation algorithms like DDPG. This is why methods like Double DQN and TD3 were ...
Jerry Ding's user avatar
1 vote
1 answer
145 views

What is the purpose of a replay/memory buffer in Deep Q-Learning networks?

I'm trying to understand DQNs. There is one concept that I cannot really understand yet. In the book "Introduction to Reinforcement Learning" as well this tutorial online introduce the ...
Dave's user avatar
  • 113
0 votes
1 answer
60 views

Trading bot with RL, automated actions, nonconvergence

I am playing around with RL to develop a trading bot (using DQN). (Disclaimer: I know, that short term stock movements are near-random and having a bot that is actually useful not likely to happen. ...
Andy's user avatar
  • 1
1 vote
1 answer
40 views

Help for my custom ENV with GYM, trying with DQN

I created a custom env to simulate a sort of blocks using gym enviroments. My environment consists of an 8 x 8 observation space, which would be the stacks of blocks and the height of each stack. <...
Betnem's user avatar
  • 11
0 votes
0 answers
26 views

Multi-Agent DQN not learning for Clean Up Game - Reward slowly decreasing

The environment of the Clean Up game is simple: in a 25*18 grid world, there's dirt spawning on the left side and apples spawning on the other. Agents get a +1 reward for eating an apple (by stepping ...
Charles's user avatar
1 vote
1 answer
10 views

DQN Loss function - doubt about stochastic approximation

In Deep Q Learning algorithm the convergence is generally achieved using smart tricks like the target network and the replay buffer. However there is one thing which is not clear to me. When the Q ...
Ftoso91's user avatar
  • 111
0 votes
0 answers
28 views

Deep Reinforcement Learning that takes action from two different sets

I am working on a problem where I want to schedule multiple activities (a1, a2, a3, ... aN) requiring different resource types ...
zeeshan's user avatar
0 votes
0 answers
19 views

Reinforcement learning: enviroment changes a lot while DQN is still computing actions

I'm implementing a agent to dodge skillshots. Here's the problem: while DQN is computing action based on observation, enviroment changed meanwhile. Since those skillshots are quite fast, the ...
口乞丿丶's user avatar
2 votes
1 answer
79 views

Filling replay buffer with expert trajectories for PPO/DQN

I have a reinforcement learning environment with sparse rewards. Current methods such as PPO and DQN both fail to learn a policy that is suffuciently good. I may have a way to find trajectories that ...
Erik Storm's user avatar
4 votes
1 answer
94 views

Finding the true Q-values in gymnaiusm

I'm very interested in the true Q-values of state-action pairs in the classic control environments in gymnasium. Contrary to the usual goal, the ordering of the Q-values itself is irrelevant; a very ...
Mark B's user avatar
  • 43
0 votes
1 answer
25 views

How does DeepQ learn for different environments?

I'm studying Deep Reinforcement Learning using the book 'DRL in Action' by Zai and Brown. In chapter 3, they present the classic GridWorld example, which can be randomly initialized. This means that ...
Hermes Morales's user avatar
0 votes
1 answer
71 views

Are All the Target Q Values in DQN same?

So I am trying to understand and make a DQN. But I didn't understand a part. So basically state's Q values computed with the network and the target Q values will also compute with a target network ...
Ege's user avatar
  • 25
0 votes
0 answers
45 views

How to tell an agent that some actions in the action space are currently not available in gym and the design of action space

I want to make a task allocation decision by reinforcement learning. Suppose there are N tasks to be allocated and M severs to serve these task. However, there is a constraint that one task should be ...
Reese's user avatar
  • 1
1 vote
1 answer
93 views

Why is my DQN agent not converging to a constant reward?

I'm currently training a DQN agent. I use an epsilon greedy exploration strategy where I decay the epsilon value linearly until it reaches 0 over 300 episodes. For the rest of the remaining 50 ...
gondorian's user avatar
1 vote
0 answers
24 views

Is there any Deep RL method that is based on value function approximation of Post-decision States

I am trying to construct an RL algorithm for managing a fleet of vehicles to maximize profit. As far as I know, the Sequential Decision Process can be decomposed into the following pic: My current ...
PokeLu's user avatar
  • 111
1 vote
0 answers
34 views

Deep Q problem. Some q-Values are always greater

Description My problem is to make actor play on stock market. I am trying to teach him playing on some portion of data. I interpolated data to uniformed interals and normalized or standarized all ...
Grzegorz Krug's user avatar
1 vote
1 answer
101 views

How would one normalize observations in off-policy online reinforcement learning?

In off-policy algorithms such as DQN, you need to feed your input to a network twice. 1. When inputting into a network for predicting the Q values. 2. When feeding the input from the buffer to the ...
desert_ranger's user avatar
2 votes
1 answer
206 views

Can I train an agent with DQN, avoiding obstacles and still finding the optimal path to finish the task?

The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible without ...
Mamora's user avatar
  • 53
3 votes
1 answer
151 views

Can DQN find optimal path while avoiding random obstacles?

Can an agent trained with a DQN algorithm in a grid world, avoid obstacles (randomly appearing during the run time) and still find the optimal path to finish a task? The agent is supposed to visit ...
Mamora's user avatar
  • 53
2 votes
1 answer
139 views

When using (s,a,r,s') to train networks could the Q network be adjusting to a suboptimal r?

My question here is that whenever you take an experience from the experience buffer (s,a,r,s') and you input r into r+yMAXQ(s',a') to get the loss. What if the r from that experience (s,a,r,s') is not ...
Stef's user avatar
  • 53
1 vote
1 answer
167 views

Deep Q Networks v Monte Carlo Tree Search in Alpha Zero

Recently I've been studying how Deep Q Networks work, and as I was reading I just assumed that game engines like Alpha Zero use Deep Q Learning to choose actions. But as I was reading the Alpha Zero ...
Kiran Manicka's user avatar
2 votes
0 answers
21 views

In DQN how does the Q network not converge to the incorrect target? [duplicate]

Whenever you are doing reinforcement learning you periodically update the target network based on the weights of the Q network. While I do understand this helps create a stable target I do not ...
Stef's user avatar
  • 53
0 votes
0 answers
26 views

Loss is negative- DQN with BCE Loss function

I am writing a code with DQN, using BCE as a loss function for the classification of a sequential time series. But while training, the loss value goes in negative. Also, accuracy and binary accuracy ...
rainarashika's user avatar
0 votes
0 answers
43 views

Training DQN Agent slows down and then at around 50 episodes

I am training a DQN Agent at around 50 episodes the fit in the replay function starts slowing down and starts freezing the PC. After a while PyCharm just crashes. This is the relevant part of the ...
willem12's user avatar
0 votes
1 answer
53 views

In DQN, how to increase epsilon but not too much?

I am using DQN algorithm in a non-stationary problem in a continuous learning. My environment gives me some new states each T steps. For example after 10 000 steps, I get some new states and I need to ...
Mouad's user avatar
  • 3
0 votes
0 answers
14 views

Training two DQNs at the same time: one for task allocation, the other for action selection

i am currently trying to tackle eternity 2 wich is a tile matching puzzle with a DDQN, but it seems to me that using only one DQN won't do the trick. I am currently using a DQN to select an action ...
mt-clemente's user avatar
0 votes
1 answer
85 views

Q learning: How to create output layer in which actions are combinations of multiple sub-actions

Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions. So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions ...
T. Kau's user avatar
  • 103
0 votes
0 answers
54 views

Problem of extremely varied reward value in DDQN

I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here. ...
Subhajit Saha's user avatar
0 votes
0 answers
88 views

Reinforcement Learning (gymnasium's FrozenLake-v1) using Spiking Neural Networks (BindsNet)

I'm new to reinforcement learning. I'm trying to solve the FrozenLake-v1 game using OpenAI's gymnasium learning environment and BindsNet, which is a library to simulate Spiking Neural Networks using ...
gthampi's user avatar
0 votes
0 answers
13 views

Request for assistance with converting legal contracts to environment for DQN

I want to convert the Extractive QA task as a Reinforcement Learning Problem Statement. So I want to integrate NLP problem into Reinforcement Learning and see if my results were achieving better when ...
Arjun Reddy's user avatar
0 votes
0 answers
66 views

DQN does not learn the CartPole problem

I've reviewed my code several times ... I can't spot the problem. When I train the agent, it doesn't learn anything. I guess the problem is the way I compute the loss function but I don't know any ...
milad heidari's user avatar
0 votes
0 answers
41 views

How to train a neural network in a RL environment to learn the optimal strategy in tic tac toe?

I have just been diving into machine learning since last week. I'm trying to create a neural network that learns to play tic tac toe optimally in an RL environment. So far with very little success. ...
Evert's user avatar
  • 101
1 vote
0 answers
101 views

What does Deep Q-Learning (DQL) do?

Hello :) I'm required to write a document where I describe what DQL does in short. This is what I wrote: DQL: instead of a Q-table, a DNN is used to approximate the Q-values for each action based on a ...
Ness's user avatar
  • 206
1 vote
1 answer
82 views

DQN with experience history to learn from already saved - which reward should I take?

I want to train a DQN model in an off-policy fashion, where my behavior policy is an older agent. I have a big memory of a lot of episodes of this agent. Now I want to find a better policy using DQN. ...
PatrickSVM's user avatar
0 votes
0 answers
219 views

Confusion about N-step buffer in DQN

So I've been trying to implement the n-step buffer in DQN algorithm. Temporal difference for which we score values $(s_i, a_i, s_{i + n}, r, d)$ is defined in this way $$ \mathrm{TD}_{n}(s_i, a_i, s^{'...
Yashiru99's user avatar
0 votes
0 answers
22 views

Is it a problem if the target network and main network are the same?

I am currently working with this code that I found on Github: this is a DQN with Graph Neural Network (GNN) on TSP problem. So, in this architecture, the target network and the main network are the ...
elizabeth's user avatar
0 votes
0 answers
23 views

Problem on evaluating DQN, on a Vehicle Routing Problem (VRP)

I am running this DQN algorithm that is trying to minimize the total distance traveled by a vehicle (VRP). In the training, as you can see in the images, everything works fine: the loss is decreasing, ...
elizabeth's user avatar
2 votes
1 answer
470 views

Should DQN/PPO be used for state spaces that are not that large?

I'm interested in trying out Q-learning to solve a problem where I already have a simulation of the environment that can run at about 100,000 fps or steps/sec. Its also continuous with no terminal ...
gameveloster's user avatar
1 vote
1 answer
117 views

In the DQN paper, why do we have both $\max_{a'}$ and $\max_{a}$ in the pseudocode?

I was reading this article https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf and in it there is an algorithm of deep q learning with experience replay as follows: On line 12, when the algorithm is ...
Ness's user avatar
  • 206
2 votes
2 answers
251 views

How are NNs output setup for games that allow multiple actions each turn and have very large sets of possible actions?

I was looking at an AI coding challenge for a two player game on a 2D grid of variable size (from one game to the next). Here is a screen shot example of the playfield. Each player has multiple units ...
snowfrogdev's user avatar
1 vote
0 answers
42 views

In what circumstances can we replace the max operator with random selection in the DQN?

In the original DQN paper, gradients during training are derived as follows: $\nabla_{\theta_i} L_i\left(\theta_i\right)=\mathbb{E}_{s, a \sim \rho(\cdot) ; s^{\prime} \sim \mathcal{E}}\left[\left(r+\...
bonzo_pippinpaddle's user avatar
0 votes
1 answer
116 views

Negative action-state values found during deep Q-learning

I'm training a simple deep q-learning algorithm with no experience buffer to solve the CartPole-v5 environment. I want to check for overestimation, therefore I'm ...
Gello's user avatar
  • 103
1 vote
0 answers
84 views

DDQN Snake keeps crashing into the wall [closed]

Edit: I managed to fix this by changing the optimizer to SGD. I am very new to reinforcement learning, and I attempted to create a DDQN for the game snake but for some reason it keeps learning to ...
ImNotKevPlayz's user avatar
1 vote
1 answer
178 views

Train Deep Q-Learning Network on a game without source code

So I have some games that I like, and I'd like to create a net that can play them, just for fun. But I don't have their source code, so I can't just pull the information I want and create a state from ...
Skyr's user avatar
  • 13
1 vote
0 answers
34 views

Replay Buffer taking long time to construct (Reinforcement Learning DQN with tf-agetns)

I'm new to Reinforcement Learning and I have some question. I am actually training some DQN using the tf-agents from tensorflow. And I recently learned that it's not possible to train a DQN using ...
M. Fire's user avatar
  • 11
1 vote
1 answer
206 views

How does the discount factor (gamma) and memory replay work in Deep Q-learning?

Maybe in over my head about this but I'm having a hard time understanding the discount factor in Deep Q-learning. Correct me if I'm wrong (1): To train a Deep Q-learning network, every N:th step of ...
DanneP's user avatar
  • 21
0 votes
1 answer
146 views

Are there better loss functions than MSE for maze solver using deep learning?

I am a newbie in reinforcement learning, and I was doing a project on solving an agent maze solver using deep Q Learning. Currently, I am using the MSE loss function, but the agent is very slow or ...
Lim's user avatar
  • 1
1 vote
2 answers
236 views

Ways to improve DQN model learning snake?

I'm training a Deep Q-learning model on a snake game and I would like some ideas on how to improve the model and maybe also efficiency of training it. The game is currently set to a 12x12 grid, a blue ...
DanneP's user avatar
  • 21
0 votes
0 answers
31 views

agent based DNN with a loopback

I have a data problem with no direct reward mechanism,(test/train) good and fault solutions. Though over a long time period good decisions might be made. I've been searching for days now for an agent ...
Peter's user avatar
  • 101

1
2 3 4 5
7