Questions tagged [dqn]
For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.
343
questions
1
vote
0
answers
12
views
Why slow-changing policy invalidates Double DQN approach in TD3 paper?
In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
- 131
2
votes
0
answers
37
views
Why does only Deep Q Learning have an overestimation bias?
There is a lot of discussion about the overestimation bias for Deep Q Learning and similar off-policy action value estimation algorithms like DDPG. This is why methods like Double DQN and TD3 were ...
- 131
1
vote
1
answer
145
views
What is the purpose of a replay/memory buffer in Deep Q-Learning networks?
I'm trying to understand DQNs.
There is one concept that I cannot really understand yet. In the book "Introduction to Reinforcement Learning" as well this tutorial online introduce the ...
- 113
0
votes
1
answer
60
views
Trading bot with RL, automated actions, nonconvergence
I am playing around with RL to develop a trading bot (using DQN). (Disclaimer: I know, that short term stock movements are near-random and having a bot that is actually useful not likely to happen. ...
- 1
1
vote
1
answer
40
views
Help for my custom ENV with GYM, trying with DQN
I created a custom env to simulate a sort of blocks using gym enviroments.
My environment consists of an 8 x 8 observation space, which would be the stacks of blocks and the height of each stack.
<...
- 11
0
votes
0
answers
26
views
Multi-Agent DQN not learning for Clean Up Game - Reward slowly decreasing
The environment of the Clean Up game is simple: in a 25*18 grid world, there's dirt spawning on the left side and apples spawning on the other. Agents get a +1 reward for eating an apple (by stepping ...
- 1
1
vote
1
answer
10
views
DQN Loss function - doubt about stochastic approximation
In Deep Q Learning algorithm the convergence is generally achieved using smart tricks like the target network and the replay buffer.
However there is one thing which is not clear to me. When the Q ...
- 111
0
votes
0
answers
28
views
Deep Reinforcement Learning that takes action from two different sets
I am working on a problem where I want to schedule multiple activities (a1, a2, a3, ... aN) requiring different resource types ...
- 1
0
votes
0
answers
19
views
Reinforcement learning: enviroment changes a lot while DQN is still computing actions
I'm implementing a agent to dodge skillshots. Here's the problem: while DQN is computing action based on observation, enviroment changed meanwhile. Since those skillshots are quite fast, the ...
- 1
2
votes
1
answer
79
views
Filling replay buffer with expert trajectories for PPO/DQN
I have a reinforcement learning environment with sparse rewards. Current methods such as PPO and DQN both fail to learn a policy that is suffuciently good. I may have a way to find trajectories that ...
- 61
4
votes
1
answer
94
views
Finding the true Q-values in gymnaiusm
I'm very interested in the true Q-values of state-action pairs in the classic control environments in gymnasium. Contrary to the usual goal, the ordering of the Q-values itself is irrelevant; a very ...
- 43
0
votes
1
answer
25
views
How does DeepQ learn for different environments?
I'm studying Deep Reinforcement Learning using the book 'DRL in Action' by Zai and Brown. In chapter 3, they present the classic GridWorld example, which can be randomly initialized. This means that ...
- 221
0
votes
1
answer
71
views
Are All the Target Q Values in DQN same?
So I am trying to understand and make a DQN. But I didn't understand a part. So basically state's Q values computed with the network and the target Q values will also compute with a target network ...
- 25
0
votes
0
answers
45
views
How to tell an agent that some actions in the action space are currently not available in gym and the design of action space
I want to make a task allocation decision by reinforcement learning. Suppose there are N tasks to be allocated and M severs to serve these task.
However, there is a constraint that one task should be ...
- 1
1
vote
1
answer
93
views
Why is my DQN agent not converging to a constant reward?
I'm currently training a DQN agent. I use an epsilon greedy exploration strategy where I decay the epsilon value linearly until it reaches 0 over 300 episodes. For the rest of the remaining 50 ...
- 45
1
vote
0
answers
24
views
Is there any Deep RL method that is based on value function approximation of Post-decision States
I am trying to construct an RL algorithm for managing a fleet of vehicles to maximize profit. As far as I know, the Sequential Decision Process can be decomposed into the following pic:
My current ...
- 111
1
vote
0
answers
34
views
Deep Q problem. Some q-Values are always greater
Description
My problem is to make actor play on stock market. I am trying to teach him playing on some portion of data. I interpolated data to uniformed interals and normalized or standarized all ...
1
vote
1
answer
101
views
How would one normalize observations in off-policy online reinforcement learning?
In off-policy algorithms such as DQN, you need to feed your input to a network twice. 1. When inputting into a network for predicting the Q values. 2. When feeding the input from the buffer to the ...
- 608
2
votes
1
answer
206
views
Can I train an agent with DQN, avoiding obstacles and still finding the optimal path to finish the task?
The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible without ...
- 53
3
votes
1
answer
151
views
Can DQN find optimal path while avoiding random obstacles?
Can an agent trained with a DQN algorithm in a grid world, avoid obstacles (randomly appearing during the run time) and still find the optimal path to finish a task? The agent is supposed to visit ...
- 53
2
votes
1
answer
139
views
When using (s,a,r,s') to train networks could the Q network be adjusting to a suboptimal r?
My question here is that whenever you take an experience from the experience buffer (s,a,r,s') and you input r into r+yMAXQ(s',a') to get the loss. What if the r from that experience (s,a,r,s') is not ...
- 53
1
vote
1
answer
167
views
Deep Q Networks v Monte Carlo Tree Search in Alpha Zero
Recently I've been studying how Deep Q Networks work, and as I was reading I just assumed that game engines like Alpha Zero use Deep Q Learning to choose actions. But as I was reading the Alpha Zero ...
2
votes
0
answers
21
views
In DQN how does the Q network not converge to the incorrect target? [duplicate]
Whenever you are doing reinforcement learning you periodically update the target network based on the weights of the Q network. While I do understand this helps create a stable target I do not ...
- 53
0
votes
0
answers
26
views
Loss is negative- DQN with BCE Loss function
I am writing a code with DQN, using BCE as a loss function for the classification of a sequential time series. But while training, the loss value goes in negative. Also, accuracy and binary accuracy ...
0
votes
0
answers
43
views
Training DQN Agent slows down and then at around 50 episodes
I am training a DQN Agent at around 50 episodes the fit in the replay function starts slowing down and starts freezing the PC. After a while PyCharm just crashes.
This is the relevant part of the ...
- 3
0
votes
1
answer
53
views
In DQN, how to increase epsilon but not too much?
I am using DQN algorithm in a non-stationary problem in a continuous learning. My environment gives me some new states each T steps. For example after 10 000 steps, I get some new states and I need to ...
- 3
0
votes
0
answers
14
views
Training two DQNs at the same time: one for task allocation, the other for action selection
i am currently trying to tackle eternity 2 wich is a tile matching puzzle with a DDQN, but it seems to me that using only one DQN won't do the trick. I am currently using a DQN to select an action ...
0
votes
1
answer
85
views
Q learning: How to create output layer in which actions are combinations of multiple sub-actions
Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions.
So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions ...
- 103
0
votes
0
answers
54
views
Problem of extremely varied reward value in DDQN
I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here.
...
- 101
0
votes
0
answers
88
views
Reinforcement Learning (gymnasium's FrozenLake-v1) using Spiking Neural Networks (BindsNet)
I'm new to reinforcement learning. I'm trying to solve the FrozenLake-v1 game using OpenAI's gymnasium learning environment and BindsNet, which is a library to simulate Spiking Neural Networks using ...
- 1
0
votes
0
answers
13
views
Request for assistance with converting legal contracts to environment for DQN
I want to convert the Extractive QA task as a Reinforcement Learning Problem Statement. So I want to integrate NLP problem into Reinforcement Learning and see if my results were achieving better when ...
- 158
0
votes
0
answers
66
views
DQN does not learn the CartPole problem
I've reviewed my code several times ... I can't spot the problem. When I train the agent, it doesn't learn anything. I guess the problem is the way I compute the loss function but I don't know any ...
0
votes
0
answers
41
views
How to train a neural network in a RL environment to learn the optimal strategy in tic tac toe?
I have just been diving into machine learning since last week. I'm trying to create a neural network that learns to play tic tac toe optimally in an RL environment. So far with very little success. ...
- 101
1
vote
0
answers
101
views
What does Deep Q-Learning (DQL) do?
Hello :) I'm required to write a document where I describe what DQL does in short.
This is what I wrote:
DQL: instead of a Q-table, a DNN is used to approximate the Q-values for each action based on a ...
- 206
1
vote
1
answer
82
views
DQN with experience history to learn from already saved - which reward should I take?
I want to train a DQN model in an off-policy fashion, where my behavior policy is an older agent. I have a big memory of a lot of episodes of this agent. Now I want to find a better policy using DQN. ...
- 33
0
votes
0
answers
219
views
Confusion about N-step buffer in DQN
So I've been trying to implement the n-step buffer in DQN algorithm.
Temporal difference for which we score values $(s_i, a_i, s_{i + n}, r, d)$ is defined in this way
$$
\mathrm{TD}_{n}(s_i, a_i, s^{'...
0
votes
0
answers
22
views
Is it a problem if the target network and main network are the same?
I am currently working with this code that I found on Github: this is a DQN with Graph Neural Network (GNN) on TSP problem.
So, in this architecture, the target network and the main network are the ...
0
votes
0
answers
23
views
Problem on evaluating DQN, on a Vehicle Routing Problem (VRP)
I am running this DQN algorithm that is trying to minimize the total distance traveled by a vehicle (VRP). In the training, as you can see in the images, everything works fine: the loss is decreasing, ...
2
votes
1
answer
470
views
Should DQN/PPO be used for state spaces that are not that large?
I'm interested in trying out Q-learning to solve a problem where I already have a simulation of the environment that can run at about 100,000 fps or steps/sec. Its also continuous with no terminal ...
- 121
1
vote
1
answer
117
views
In the DQN paper, why do we have both $\max_{a'}$ and $\max_{a}$ in the pseudocode?
I was reading this article https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf and in it there is an algorithm of deep q learning with experience replay as follows:
On line 12, when the algorithm is ...
- 206
2
votes
2
answers
251
views
How are NNs output setup for games that allow multiple actions each turn and have very large sets of possible actions?
I was looking at an AI coding challenge for a two player game on a 2D grid of variable size (from one game to the next).
Here is a screen shot example of the playfield.
Each player has multiple units ...
- 121
1
vote
0
answers
42
views
In what circumstances can we replace the max operator with random selection in the DQN?
In the original DQN paper, gradients during training are derived as follows:
$\nabla_{\theta_i} L_i\left(\theta_i\right)=\mathbb{E}_{s, a \sim \rho(\cdot) ; s^{\prime} \sim \mathcal{E}}\left[\left(r+\...
0
votes
1
answer
116
views
Negative action-state values found during deep Q-learning
I'm training a simple deep q-learning algorithm with no experience buffer to solve the CartPole-v5 environment.
I want to check for overestimation, therefore I'm ...
- 103
1
vote
0
answers
84
views
DDQN Snake keeps crashing into the wall [closed]
Edit:
I managed to fix this by changing the optimizer to SGD.
I am very new to reinforcement learning, and I attempted to create a DDQN for the game snake but for some reason it keeps learning to ...
1
vote
1
answer
178
views
Train Deep Q-Learning Network on a game without source code
So I have some games that I like, and I'd like to create a net that can play them, just for fun. But I don't have their source code, so I can't just pull the information I want and create a state from ...
- 13
1
vote
0
answers
34
views
Replay Buffer taking long time to construct (Reinforcement Learning DQN with tf-agetns)
I'm new to Reinforcement Learning and I have some question. I am actually training some DQN using the tf-agents from tensorflow. And I recently learned that it's not possible to train a DQN using ...
- 11
1
vote
1
answer
206
views
How does the discount factor (gamma) and memory replay work in Deep Q-learning?
Maybe in over my head about this but I'm having a hard time understanding the discount factor in Deep Q-learning.
Correct me if I'm wrong (1):
To train a Deep Q-learning network, every N:th step of ...
- 21
0
votes
1
answer
146
views
Are there better loss functions than MSE for maze solver using deep learning?
I am a newbie in reinforcement learning, and I was doing a project on solving an agent maze solver using deep Q Learning. Currently, I am using the MSE loss function, but the agent is very slow or ...
- 1
1
vote
2
answers
236
views
Ways to improve DQN model learning snake?
I'm training a Deep Q-learning model on a snake game and I would like some ideas on how to improve the model and maybe also efficiency of training it.
The game is currently set to a 12x12 grid, a blue ...
- 21
0
votes
0
answers
31
views
agent based DNN with a loopback
I have a data problem with no direct reward mechanism,(test/train) good and fault solutions.
Though over a long time period good decisions might be made.
I've been searching for days now for an agent ...
- 101