Questions tagged [dqn]

For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.

Filter by
Sorted by
Tagged with
3
votes
0answers
144 views

DQN Agent not learning anymore - what can I do to fix this?

I am trying to use Deep-Q-Learning to learn an ANN which controls a 7-DOF robotic arm. The robotic arm must avoid an obstacle and reach a target. I have implemented a number of state-of-art ...
0
votes
1answer
57 views

Encourage Deep Q to seek short-term reward

I understand that gamma is an important factor in determining the rewards for a deep Q agent, however during testing of my network I am noticing that the agent is outputting more actions to "do ...
1
vote
0answers
37 views

Deciding the rewards for different actions in Pong for a DQN agent

I am attempting to implement an agent that learns to play in the Pong environment, the environment was created in PyGame and I return the pixel data and score at each frame. I use a CNN to take a ...
5
votes
2answers
310 views

Why don't people use projected Bellman error with deep neural networks?

Projected Bellman error has shown to be stable with linear function approximation. The technique is not at all new. I can only wonder why this technique is not adopted to use with non-linear function ...
3
votes
0answers
158 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
5
votes
2answers
258 views

Can DQN perform better than Double DQN?

I'm training both kind of agents against an environment but DQN performs significantly better than Double DQN. As I've saw here, Double DQN use to perform better than DQN. Am I doing something wrong ...
2
votes
1answer
34 views

Why do DQNs use linear activations on cartpole?

I've been reading a lot of tutorials on DQNs for cartpole. In many of them, they have the funnel layer of the neural net be a linear activation. Why is this? Is it just a choice made by the ...
1
vote
2answers
161 views

Can gamma be greater than 1 in a DQN?

If I have a DQN, and I care A LOT about future rewards (moreso than current rewards), can I set gamma to a number greater than 1? Like 1.1 perhaps?
1
vote
1answer
64 views

DQN Q-values are static

I am working on a DDQN with 5 LSTM layers and 3 actions as output and state space of 21 features. I am dividing the dataset into episodes of 720 timesteps, for each episode the agent acts greedily for ...
0
votes
0answers
34 views

Next step after lane detection in vehicle automation

I have an RC car with a camera, I have implemented so that i can detect lanes on my track (think like a nascar track). I want to get this car to be able to go around the track autonomous. But I am ...
0
votes
0answers
293 views

DQN Q-mean values converge negatively

I'm trying to implement my own DQN. So far I think my code is good, but my Q-values (I'm getting the mean of all the values for every episode) tends to converge near-zero but negatively. It is normal? ...
1
vote
1answer
62 views

How to build a DQN agent which can be trained through interactive learning?

I am trying to create a chatbot whose dialogue policy model will be trained through reinforcement learning. Dialogue Policy is responsible for selecting the action to take based on the given state of ...
1
vote
0answers
83 views

Do we need to reset the DQN network after every episode?

I was going through this implementation of Reinforcement learning where model is being trained to manage the number of bikes at a station. Here, line 78 represents the loop over all episodes (if I ...
3
votes
1answer
97 views

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

Twin Delayed Deep Deterministic (TD3) policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained ...
3
votes
1answer
305 views

What are the differences between the DQN variants?

There are several variants of the DQN model. For example, double DQN, duelling DQN, prioritized DQN, distributed prioritized DQN, episodic memory DQN, asynchronous n-step DQN and multiple DQN. What ...
1
vote
0answers
37 views

Comparison and understanding of different version of DDQN?

There are several version of DDQN floating around. Sutton gives one that is a simple symmetric random update of the two Q functions. I think other papers (Silver paper for example) use a kind of ...
1
vote
1answer
90 views

OpenAI-Gym excess of actions

I'm trying to replicate the DeepMind DQN paper, and actually I'm using the OpenAI-Gym enviroment. I'm trying to get a decent score with Space Invaders (using ...
0
votes
0answers
103 views

Having trouble solving Pong. My model is not improving

Im trying to solve pong by a DQN approach. These are my hyper parameters: ...
8
votes
3answers
2k views

Huge action space size in Reinforcement Learning

I am working on creating a RL based AI for a certain board game. Just as a general overview of the game so that you understand what it's all about: It's a discrete turn-based game with a board of size ...
2
votes
1answer
2k views

In DQN, updating target network every N steps or slowly update every step is better?

The use of target network is to reduce the chance of value divergence which could happen with off-policy samples trained with semi-gradient objectives. In Deep Q network, semi-gradient TD is used and ...
5
votes
2answers
954 views

What is the difference between DQN and AlphaGo Zero?

I have already implemented a relatively simple DQN on Pacman. Now I would like to clearly understand the difference between a DQN and the techniques used by AlphaGo zero/AlphaZero and I couldn't find ...
3
votes
2answers
951 views

My DQN is stuck and can't see where the problem is

I'm trying to replicate the DeepMind paper results, so I implemented my own DQN. I left it training for more than 4 million frames (more than 2000 episodes) on SpaceInvaders-v4 (OpenAI-Gym) and it ...
3
votes
2answers
967 views

Each training run for DDQN agent takes 2 days, and still ends up with -13 avg score, but OpenAi baseline DQN needs only an hour to converge to +18?

Status: For a few weeks now, I have been working on a Double DQN agent for the PongDeterministic-v4 environment, which you can find here. A single training run ...
0
votes
3answers
170 views

For some reasons, a reward becomes a penalty if

I am working to build a reinforcement agent with DQN. The agent would be able to place buy and sell orders for a day trading purpose. I am facing a little problem with that project. The question is "...
3
votes
2answers
544 views

What does it mean by high dimensional state in DQN?

Going through the DQN paper, it said the state-space is high dimensional. I am a little bit confused here. Suppose my state is a high dimensional vector of N length ...
1
vote
1answer
414 views

DQN exploration strategy for large grid-world environment

My task involves a large grid-world type of environment (grid size may be $30\times30$, $50\times50$, $100\times100$, at the largest $200\times200$). Each element in this grid either contains a 0 or a ...
1
vote
0answers
67 views

Exploration rate decay and training in Q learning

I'm trying to replicate the results of the DeepMind's paper with Breakout included in OpenAI Gym. I wonder how much frames should I keep until I reach the fixed exploration rate. Actually it reaches ...
0
votes
1answer
696 views

DQN it's not working properly

I'm trying to build a DQN to replicate the DeepMind results. I'm doing with a simple DQN for the moment, but it isn't learning properly: after +5000 episodes, it couldn't get more than 9-10 points. ...
1
vote
1answer
529 views

DQN Breakout adding an extra negative reward to help training?

I'm trying to train a DQN, so I'm using OpenAI gym and Breakout (Breakout-v0). I have altered the reward supplied by the environment: If the episode is not completed fully, the agent gets a -10 ...
3
votes
1answer
165 views

Reason for issues with correlation in the dataset in DQN

From the paper Human level Control through DeepRL, the correlation in the data causes instability in the network and may causes the network to ...
4
votes
1answer
902 views

Ensure convergence of DDQN if true Q-values are very close

I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future reward from ...
1
vote
1answer
164 views

Using a DQN with a variable amount of Valid Moves per turn for a Board Game

I have created a game on an 8x8 grid and there are 4 pieces which can move essentially like checkers pieces (Forward left or Forward right only). I have implemented a DQN in order to pull this off. ...
2
votes
1answer
89 views

Can the opponent's turn affect the reward for a DQN agent action?

I made an engine for a 2 players card game and now I am trying to make an environment similar to OpenAI Gym envs, to ease out the training. I fail to understand this thing however: If I use ...
2
votes
1answer
2k views

Is Deep Q Neural Network (DQN) applicable only with images as inputs?

More precisely: is DQNN applicable only when we have high translational invariance in our input(s)? Starting from the original paper on nature (here a version stored on googleapis) and after looking ...
2
votes
1answer
107 views

Can DQN announce it has things in its hand in a card game?

More informations on the card game I'm talking about are in my last question here: DQN input representation for a card game So I was thinking about the output of the q neural network and, aside from ...
2
votes
1answer
87 views

Convergence in multi-agent environment

I have a multi-agent environment where agents are trying to optimise the overall energy consumption of their group. Agents can exchange energy between themselves (actions for exchange of energy ...
3
votes
1answer
1k views

DQN input representation for a card game

In order to learn about DP and RL, I chose to start a side project where I would train an AI to play a "simple" card game. I will be doing this using the DQN with replay memory. The problem is, I can'...
14
votes
1answer
2k views

Why does DQN require two different networks?

I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...
3
votes
1answer
1k views

How should I model all available actions of a chess game in deep Q-learning?

I just read about deep Q-learning, which is using a neural network for the value function instead of a table. I saw the example here: https://yanpanlau.github.io/2016/07/10/FlappyBird-Keras.html and ...
1
vote
0answers
53 views

Deep Q-Network concepts and implementation

How does sequential DQN work? How would one construct the simple sequential DQN? OpenAI Baselines: DQN
3
votes
1answer
745 views

Why use semi-gradient instead of full gradient in RL problems, when using function approximation?

Semi-gradient methods work well in reinforcement learning, but what is there a reason of not using the true gradient if it can be computed? I tried it on the cart pole problem with a deep Q-network ...

1 2 3
4