Questions tagged [dqn]

For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.

Filter by
Sorted by
Tagged with
1 vote
0 answers
12 views

How to measure accuracy of learned value function of a fixed policy?

Let's say we've a given policy whose value function is to be evaluated. One way to get the value function can be using expected SARSA, as in this stack exchange answer. However, my MDP's state space ...
ModCon's user avatar
  • 111
0 votes
2 answers
38 views

Can DQN lead with discrete state spaces?

For example in Cart Pole v1 gym environment the state space is continuous, but we discretize it to apply the Q-Learning algorithm because Q-Learning is a tabular method and only works with discrete ...
Vitor Martins's user avatar
3 votes
1 answer
169 views

Does the DoubleDQN algorithm use a target network or two separate policies?

I've been looking for ways to improve my DQN. That is when I found the Double DQN algorithm. After looking at explanatory videos and posts, I've seen conflicting information: The Double DQN algorithm ...
Vladislav Korecký's user avatar
2 votes
1 answer
143 views

Q learning (DQN) strategy for a multiplayer zero-sum game

I have been looking for ways to train a Q-learning agent for a multiplayer zero-sum game (a variation of Tic-Tac-Toe in my case). I came up with a learning strategy I haven't found anywhere else, and ...
Vladislav Korecký's user avatar
0 votes
1 answer
25 views

Can/should a reward function depend on something other than state in a DQN

Question: Is it OK to have a reward function on a DQN or any RL algorithm that depends on variables other than the enviroment state? I'm asking because, so far I'm learning from tutorials, but I've ...
Oliver Mohr Bonometti's user avatar
0 votes
1 answer
36 views

CNN Input shape for DQN Q-calculating Network

Context: I want to build a DQN with as CNN for calculating its Q value on each step. Enviroment's status can be described by the attributes of 3 machines (each one with own attributes). I'd also like ...
Oliver Mohr Bonometti's user avatar
0 votes
0 answers
21 views

Summing up rewards when encountering terminal state in n-step DQN

I'm trying to implement n-step DQN using deque for n-step experience buffer and I am not sure how to handle the terminal state in calculation For step = n = 5, Sx = state number x, T - terminal state, ...
Question's user avatar
0 votes
0 answers
20 views

Reinforcement learning for a word game

Let's imagine a simple word game where there is a grid of letters. The agent starts at a letter and moves to a neighboring letter, continuing in this fashion to form a word. For instance in this grid ...
mustafa's user avatar
0 votes
0 answers
49 views

How do I deal with dynamic, parameterized, action spaces?

I want to design an AI Learning Algorithm for a Student made, round based Game. Let me first explain the Game/Environment You have a round based HTTP Game, in which multiple Players can participate. ...
Andre's user avatar
  • 1
0 votes
0 answers
15 views

how to choose the reward function to solve an optimization problem with DQN

I am currently working on solving an optimization problem related to the facilities layout using Deep Q-Networks (DQN). The primary challenge I am facing revolves around designing an effective reward ...
CHIBOUB Amine's user avatar
0 votes
1 answer
21 views

Relationship between regularization and (effective) discounting in deep Q learning

I have a deep-Q-network-type reinforcement learner in a minigrid-type environment. After learning I can put the agent in a series of contrived situations and measure its Q values, and then infer its ...
dinnums's user avatar
1 vote
1 answer
63 views

How is state space normalization done in off-policy algorithms like dqn? [closed]

There are 4 features in my state representation and they are in different ranges. So I'm thinking state normalization would reduce the bias on certain features. And also, in the problem I consider, ...
Aki A's user avatar
  • 11
5 votes
2 answers
281 views

DQN arXiv 10-year anniversary: What are the outstanding problems being actively researched in deep Q-learning since 2019?

Background As of today (12-19-2023), the arXiv submission of the original deep Q-learning approach to achieve superhuman performance on ATARI games has turned a decade old. The original approach, ...
DeepQZero's user avatar
  • 1,379
1 vote
0 answers
29 views

Why slow-changing policy invalidates Double DQN approach in TD3 paper?

In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
Jerry Ding's user avatar
2 votes
0 answers
83 views

Why does only Deep Q Learning have an overestimation bias?

There is a lot of discussion about the overestimation bias for Deep Q Learning and similar off-policy action value estimation algorithms like DDPG. This is why methods like Double DQN and TD3 were ...
Jerry Ding's user avatar
1 vote
1 answer
463 views

What is the purpose of a replay/memory buffer in Deep Q-Learning networks?

I'm trying to understand DQNs. There is one concept that I cannot really understand yet. In the book "Introduction to Reinforcement Learning" as well this tutorial online introduce the ...
Dave's user avatar
  • 155
0 votes
1 answer
156 views

Trading bot with RL, automated actions, nonconvergence

I am playing around with RL to develop a trading bot (using DQN). (Disclaimer: I know, that short term stock movements are near-random and having a bot that is actually useful not likely to happen. ...
Andy's user avatar
  • 1
1 vote
1 answer
46 views

Help for my custom ENV with GYM, trying with DQN

I created a custom env to simulate a sort of blocks using gym enviroments. My environment consists of an 8 x 8 observation space, which would be the stacks of blocks and the height of each stack. <...
Betnem's user avatar
  • 11
0 votes
0 answers
37 views

Multi-Agent DQN not learning for Clean Up Game - Reward slowly decreasing

The environment of the Clean Up game is simple: in a 25*18 grid world, there's dirt spawning on the left side and apples spawning on the other. Agents get a +1 reward for eating an apple (by stepping ...
Charles's user avatar
1 vote
1 answer
21 views

DQN Loss function - doubt about stochastic approximation

In Deep Q Learning algorithm the convergence is generally achieved using smart tricks like the target network and the replay buffer. However there is one thing which is not clear to me. When the Q ...
Ftoso91's user avatar
  • 111
0 votes
0 answers
32 views

Deep Reinforcement Learning that takes action from two different sets

I am working on a problem where I want to schedule multiple activities (a1, a2, a3, ... aN) requiring different resource types ...
zeeshan's user avatar
0 votes
0 answers
21 views

Reinforcement learning: enviroment changes a lot while DQN is still computing actions

I'm implementing a agent to dodge skillshots. Here's the problem: while DQN is computing action based on observation, enviroment changed meanwhile. Since those skillshots are quite fast, the ...
口乞丿丶's user avatar
2 votes
1 answer
223 views

Filling replay buffer with expert trajectories for PPO/DQN

I have a reinforcement learning environment with sparse rewards. Current methods such as PPO and DQN both fail to learn a policy that is suffuciently good. I may have a way to find trajectories that ...
Erik Storm's user avatar
4 votes
1 answer
101 views

Finding the true Q-values in gymnaiusm

I'm very interested in the true Q-values of state-action pairs in the classic control environments in gymnasium. Contrary to the usual goal, the ordering of the Q-values itself is irrelevant; a very ...
Mark B's user avatar
  • 43
0 votes
1 answer
26 views

How does DeepQ learn for different environments?

I'm studying Deep Reinforcement Learning using the book 'DRL in Action' by Zai and Brown. In chapter 3, they present the classic GridWorld example, which can be randomly initialized. This means that ...
Hermes Morales's user avatar
0 votes
1 answer
112 views

Are All the Target Q Values in DQN same?

So I am trying to understand and make a DQN. But I didn't understand a part. So basically state's Q values computed with the network and the target Q values will also compute with a target network ...
Ege's user avatar
  • 25
0 votes
0 answers
66 views

How to tell an agent that some actions in the action space are currently not available in gym and the design of action space

I want to make a task allocation decision by reinforcement learning. Suppose there are N tasks to be allocated and M severs to serve these task. However, there is a constraint that one task should be ...
Reese's user avatar
  • 1
1 vote
1 answer
207 views

Why is my DQN agent not converging to a constant reward?

I'm currently training a DQN agent. I use an epsilon greedy exploration strategy where I decay the epsilon value linearly until it reaches 0 over 300 episodes. For the rest of the remaining 50 ...
gondorian's user avatar
1 vote
0 answers
26 views

Is there any Deep RL method that is based on value function approximation of Post-decision States

I am trying to construct an RL algorithm for managing a fleet of vehicles to maximize profit. As far as I know, the Sequential Decision Process can be decomposed into the following pic: My current ...
PokeLu's user avatar
  • 111
1 vote
0 answers
35 views

Deep Q problem. Some q-Values are always greater

Description My problem is to make actor play on stock market. I am trying to teach him playing on some portion of data. I interpolated data to uniformed interals and normalized or standarized all ...
Grzegorz Krug's user avatar
1 vote
1 answer
147 views

How would one normalize observations in off-policy online reinforcement learning?

In off-policy algorithms such as DQN, you need to feed your input to a network twice. 1. When inputting into a network for predicting the Q values. 2. When feeding the input from the buffer to the ...
desert_ranger's user avatar
2 votes
1 answer
243 views

Can I train an agent with DQN, avoiding obstacles and still finding the optimal path to finish the task?

The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible without ...
Mamora's user avatar
  • 53
3 votes
1 answer
166 views

Can DQN find optimal path while avoiding random obstacles?

Can an agent trained with a DQN algorithm in a grid world, avoid obstacles (randomly appearing during the run time) and still find the optimal path to finish a task? The agent is supposed to visit ...
Mamora's user avatar
  • 53
2 votes
1 answer
139 views

When using (s,a,r,s') to train networks could the Q network be adjusting to a suboptimal r?

My question here is that whenever you take an experience from the experience buffer (s,a,r,s') and you input r into r+yMAXQ(s',a') to get the loss. What if the r from that experience (s,a,r,s') is not ...
Stef's user avatar
  • 95
1 vote
1 answer
389 views

Deep Q Networks v Monte Carlo Tree Search in Alpha Zero

Recently I've been studying how Deep Q Networks work, and as I was reading I just assumed that game engines like Alpha Zero use Deep Q Learning to choose actions. But as I was reading the Alpha Zero ...
Kiran Manicka's user avatar
2 votes
0 answers
21 views

In DQN how does the Q network not converge to the incorrect target? [duplicate]

Whenever you are doing reinforcement learning you periodically update the target network based on the weights of the Q network. While I do understand this helps create a stable target I do not ...
Stef's user avatar
  • 95
0 votes
0 answers
39 views

Loss is negative- DQN with BCE Loss function

I am writing a code with DQN, using BCE as a loss function for the classification of a sequential time series. But while training, the loss value goes in negative. Also, accuracy and binary accuracy ...
rainarashika's user avatar
0 votes
0 answers
47 views

Training DQN Agent slows down and then at around 50 episodes

I am training a DQN Agent at around 50 episodes the fit in the replay function starts slowing down and starts freezing the PC. After a while PyCharm just crashes. This is the relevant part of the ...
willem12's user avatar
0 votes
1 answer
88 views

In DQN, how to increase epsilon but not too much?

I am using DQN algorithm in a non-stationary problem in a continuous learning. My environment gives me some new states each T steps. For example after 10 000 steps, I get some new states and I need to ...
Mouad's user avatar
  • 3
0 votes
1 answer
105 views

Q learning: How to create output layer in which actions are combinations of multiple sub-actions

Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions. So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions ...
T. Kau's user avatar
  • 103
0 votes
0 answers
64 views

Problem of extremely varied reward value in DDQN

I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here. ...
Subhajit Saha's user avatar
0 votes
0 answers
13 views

Request for assistance with converting legal contracts to environment for DQN

I want to convert the Extractive QA task as a Reinforcement Learning Problem Statement. So I want to integrate NLP problem into Reinforcement Learning and see if my results were achieving better when ...
Arjun Reddy's user avatar
1 vote
0 answers
126 views

What does Deep Q-Learning (DQL) do?

Hello :) I'm required to write a document where I describe what DQL does in short. This is what I wrote: DQL: instead of a Q-table, a DNN is used to approximate the Q-values for each action based on a ...
Ness's user avatar
  • 206
1 vote
1 answer
99 views

DQN with experience history to learn from already saved - which reward should I take?

I want to train a DQN model in an off-policy fashion, where my behavior policy is an older agent. I have a big memory of a lot of episodes of this agent. Now I want to find a better policy using DQN. ...
PatrickSVM's user avatar
2 votes
1 answer
658 views

Should DQN/PPO be used for state spaces that are not that large?

I'm interested in trying out Q-learning to solve a problem where I already have a simulation of the environment that can run at about 100,000 fps or steps/sec. Its also continuous with no terminal ...
gameveloster's user avatar
1 vote
1 answer
124 views

In the DQN paper, why do we have both $\max_{a'}$ and $\max_{a}$ in the pseudocode?

I was reading this article https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf and in it there is an algorithm of deep q learning with experience replay as follows: On line 12, when the algorithm is ...
Ness's user avatar
  • 206
2 votes
2 answers
300 views

How are NNs output setup for games that allow multiple actions each turn and have very large sets of possible actions?

I was looking at an AI coding challenge for a two player game on a 2D grid of variable size (from one game to the next). Here is a screen shot example of the playfield. Each player has multiple units ...
snowfrogdev's user avatar
1 vote
0 answers
45 views

In what circumstances can we replace the max operator with random selection in the DQN?

In the original DQN paper, gradients during training are derived as follows: $\nabla_{\theta_i} L_i\left(\theta_i\right)=\mathbb{E}_{s, a \sim \rho(\cdot) ; s^{\prime} \sim \mathcal{E}}\left[\left(r+\...
bonzo_pippinpaddle's user avatar
0 votes
1 answer
174 views

Negative action-state values found during deep Q-learning

I'm training a simple deep q-learning algorithm with no experience buffer to solve the CartPole-v5 environment. I want to check for overestimation, therefore I'm ...
Gello's user avatar
  • 103
1 vote
0 answers
94 views

DDQN Snake keeps crashing into the wall [closed]

Edit: I managed to fix this by changing the optimizer to SGD. I am very new to reinforcement learning, and I attempted to create a DDQN for the game snake but for some reason it keeps learning to ...
ImNotKevPlayz's user avatar

1
2 3 4 5
7