Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [dqn]

Deep Q-Network. Can be used in conjunction with the "q-learning" tag or independently.

3
votes
1answer
31 views

How do I represent a multi-dimensional state using a neural network?

I have a set of 15 unique playing cards from a deck of 52 playing cards. A given state is represented by the respective card values in the set of 15 cards, where the card value is a prime number ...
0
votes
0answers
13 views

DQN Algorithm not Converging for HVAC Control Task

I am trying to implement the DQN algorithm for the task of HVAC control. I have the algorithm implemented using Pytorch. I know that the HVAC simulator is working as it works for other control methods....
3
votes
2answers
58 views

Deciding on a reward per each action in a given state (Q-learning)

I looked for existing posts on Stack Exchange, which kind of answer the questions about the reward system and reward function, but not specifically what I want to ask here, which is how do you ...
1
vote
0answers
12 views

DQN ANN input vs Linear function approximator feature vector

So when using semi-gradient td(0) you need to convert your state representation into a feature vector that represents the state and as far as I know, should not be correlated. Is the input on the ANN ...
3
votes
1answer
27 views

Which kind of prioritized experience replay should I use?

The Prioritized Experience Replay paper gives two different ways of sampling from the replay buffer. One, called "proportional prioritization", assigns each transition a priority proportional to its ...
2
votes
1answer
46 views

Experience Replay Not Always Giving Better Results

I have recently started working on a control problem using a Deep Q Network as proposed by DeepMind (https://arxiv.org/abs/1312.5602). Initially, I implemented it without Experience Replay. The ...
1
vote
3answers
44 views

How to stop DQN Q function from increasing during learning?

Following the DQN algorithm with experience replay: We calculate the $loss=(Q(s,a)-(r+Q(s+1,a)))^2$. Assume I have positive but changing rewards. Meaning, $r>0$. Thus, since the rewards are ...
2
votes
0answers
21 views

Why experience reply memory in DQN instead of a RNN memory?

I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
3
votes
0answers
32 views

DQN Agent not learning anymore - what can I do to fix this?

I am trying to use Deep-Q-Learning to learn an ANN which controls a 7-DOF robotic arm. The robotic arm must avoid an obstacle and reach a target. I have implemented a number of state-of-art ...
0
votes
1answer
36 views

Encourage Deep Q to seek short-term reward

I understand that gamma is an important factor in determining the rewards for a deep Q agent, however during testing of my network I am noticing that the agent is outputting more actions to "do ...
1
vote
0answers
17 views

Deciding the rewards for different actions in Pong for a DQN agent

I am attempting to implement an agent that learns to play in the Pong environment, the environment was created in PyGame and I return the pixel data and score at each frame. I use a CNN to take a ...
3
votes
0answers
44 views

Why don't people use projected Bellman error with deep neural networks?

Projected Bellman error has shown to be stable with linear function approximation. The technique is not at all new. I can only wonder why this technique is not adopted to use with non-linear function ...
3
votes
0answers
70 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
4
votes
2answers
63 views

Can DQN perform better than Double DQN?

I'm training both kind of agents against an environment but DQN performs significantly better than Double DQN. As I've saw here, Double DQN use to perform better than DQN. Am I doing something wrong ...
2
votes
1answer
21 views

Why do DQNs use linear activations on cartpole?

I've been reading a lot of tutorials on DQNs for cartpole. In many of them, they have the funnel layer of the neural net be a linear activation. Why is this? Is it just a choice made by the ...
1
vote
2answers
37 views

Can gamma be greater than 1 in a DQN?

If I have a DQN, and I care A LOT about future rewards (moreso than current rewards), can I set gamma to a number greater than 1? Like 1.1 perhaps?
1
vote
1answer
38 views

DQN Q-values are static

I am working on a DDQN with 5 LSTM layers and 3 actions as output and state space of 21 features. I am dividing the dataset into episodes of 720 timesteps, for each episode the agent acts greedily for ...
0
votes
0answers
14 views

Next step after lane detection in vehicle automation

I have an RC car with a camera, I have implemented so that i can detect lanes on my track (think like a nascar track). I want to get this car to be able to go around the track autonomous. But I am ...
0
votes
0answers
16 views

How to limit actions based on a state [duplicate]

I'm trying to implement DQN using tf-agents for simple environment. So far I have ...
0
votes
0answers
57 views

DQN Q-mean values converge negatively

I'm trying to implement my own DQN. So far I think my code is good, but my Q-values (I'm getting the mean of all the values for every episode) tends to converge near-zero but negatively. It is normal? ...
1
vote
1answer
43 views

How to build a DQN agent which can be trained through interactive learning?

I am trying to create a chatbot whose dialogue policy model will be trained through reinforcement learning. Dialogue Policy is responsible for selecting the action to take based on the given state of ...
1
vote
0answers
43 views

Do we need to reset the DQN network after every episode?

I was going through this implementation of Reinforcement learning where model is being trained to manage the number of bikes at a station. Here, line 78 represents the loop over all episodes (if I ...
2
votes
1answer
51 views

What are the differences between the DQN variants?

There are several variants of the DQN model. For example, double DQN, duelling DQN, prioritized DQN, distributed prioritized DQN, episodic memory DQN, asynchronous n-step DQN and multiple DQN. What ...
1
vote
0answers
27 views

Comparison and understanding of different version of DDQN?

There are several version of DDQN floating around. Sutton gives one that is a simple symmetric random update of the two Q functions. I think other papers (Silver paper for example) use a kind of ...
0
votes
1answer
31 views

OpenAI-Gym excess of actions

I'm trying to replicate the DeepMind DQN paper, and actually I'm using the OpenAI-Gym enviroment. I'm trying to get a decent score with Space Invaders (using ...
0
votes
0answers
31 views

Having trouble solving Pong. My model is not improving

Im trying to solve pong by a DQN approach. These are my hyper parameters: ...
1
vote
1answer
94 views

Huge action space size in Reinforcement Learning

I am working on creating a RL based AI for a certain board game. Just as a general overview of the game so that you understand what it's all about: It's a discrete turn-based game with a board of size ...
2
votes
1answer
123 views

In DQN, updating target network every N steps or slowly update every step is better?

The use of target network is to reduce the chance of value divergence which could happen with off-policy samples trained with semi-gradient objectives. In Deep Q network, semi-gradient TD is used and ...
4
votes
2answers
177 views

What is the difference between DQN and AlphaGo Zero?

I have already implemented a relatively simple DQN on Pacman. Now I would like to clearly understand the difference between a DQN and the techniques used by AlphaGo zero/AlphaZero and I couldn't find ...
3
votes
2answers
133 views

My DQN is stuck and can't see where the problem is

I'm trying to replicate the DeepMind paper results, so I implemented my own DQN. I left it training for more than 4 million frames (more than 2000 episodes) on SpaceInvaders-v4 (OpenAI-Gym) and it ...
3
votes
2answers
150 views

Each training run for DDQN agent takes 2 days, and still ends up with -13 avg score, but OpenAi baseline DQN needs only an hour to converge to +18?

Status: For a few weeks now, I have been working on a Double DQN agent for the PongDeterministic-v4 environment, which you can find here. A single training run ...
0
votes
3answers
141 views

For some reasons, a reward becomes a penalty if

I am working to build a reinforcement agent with DQN. The agent would be able to place buy and sell orders for a day trading purpose. I am facing a little problem with that project. The question is "...
1
vote
2answers
77 views

What does it mean by high dimensional state in DQN?

Going through the DQN paper, it said the state-space is high dimensional. I am a little bit confused here. Suppose my state is a high dimensional vector of N length ...
0
votes
1answer
124 views

DQN exploration strategy for large grid-world environment

My task involves a large grid-world type of environment (grid size may be $30\times30$, $50\times50$, $100\times100$, at the largest $200\times200$). Each element in this grid either contains a 0 or a ...
1
vote
0answers
29 views

Exploration rate decay and training in Q learning

I'm trying to replicate the results of the DeepMind's paper with Breakout included in OpenAI Gym. I wonder how much frames should I keep until I reach the fixed exploration rate. Actually it reaches ...
-1
votes
1answer
236 views

DQN it's not working properly

I'm trying to build a DQN to replicate the DeepMind results. I'm doing with a simple DQN for the moment, but it isn't learning properly: after +5000 episodes, it couldn't get more than 9-10 points. ...
1
vote
1answer
155 views

DQN Breakout adding an extra negative reward to help training?

I'm trying to train a DQN, so I'm using OpenAI gym and Breakout (Breakout-v0). I have altered the reward supplied by the environment: If the episode is not completed fully, the agent gets a -10 ...
3
votes
1answer
53 views

Reason for issues with correlation in the dataset in DQN

From the paper Human level Control through DeepRL, the correlation in the data causes instability in the network and may causes the network to ...
2
votes
1answer
277 views

Ensure convergence of DDQN if true Q-values are very close

I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future reward from ...
1
vote
1answer
80 views

Using a DQN with a variable amount of Valid Moves per turn for a Board Game

I have created a game on an 8x8 grid and there are 4 pieces which can move essentially like checkers pieces (Forward left or Forward right only). I have implemented a DQN in order to pull this off. ...
0
votes
0answers
70 views

What is a copy of network to create a “target network” for in DQN? [duplicate]

I'm now reading a book titled "Hands-On Reinforcement Learning with Python" by O'Reilly, and the author said the following to implement the DQN algorithm. To make training more stable, there is a ...
2
votes
1answer
59 views

Can the opponent's turn affect the reward for a DQN agent action?

I made an engine for a 2 players card game and now I am trying to make an environment similar to OpenAI Gym envs, to ease out the training. I fail to understand this thing however: If I use ...
2
votes
1answer
770 views

Is Deep Q Neural Network (DQN) applicable only with images as inputs?

More precisely: is DQNN applicable only when we have high translational invariance in our input(s)? Starting from the original paper on nature (here a version stored on googleapis) and after looking ...
2
votes
1answer
71 views

Can DQN announce it has things in its hand in a card game?

More informations on the card game I'm talking about are in my last question here: DQN input representation for a card game So I was thinking about the output of the q neural network and, aside from ...
2
votes
1answer
59 views

Convergence in multi-agent environment

I have a multi-agent environment where agents are trying to optimise the overall energy consumption of their group. Agents can exchange energy between themselves (actions for exchange of energy ...
2
votes
1answer
465 views

DQN input representation for a card game

In order to learn about DP and RL, I chose to start a side project where I would train an AI to play a "simple" card game. I will be doing this using the DQN with replay memory. The problem is, I can'...
11
votes
1answer
1k views

Why does DQN require two different networks?

I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...
1
vote
0answers
50 views

Deep Q-Network concepts and implementation

How does sequential DQN work? How would one construct the simple sequential DQN? OpenAI Baselines: DQN
1
vote
1answer
430 views

Why use semi-gradient instead of full gradient in RL problems, when using function approximation?

Semi-gradient methods work well in reinforcement learning, but what is there a reason of not using the true gradient if it can be computed? I tried it on the cart pole problem with a deep Q-network ...