Questions tagged [dqn]

For questions related to the deep Q-network (DQN), which is a deep neural network (e.g. a convolutional neural network) trained with a variant of Q-learning. The expression was coined in the paper "Playing Atari with Deep Reinforcement Learning" (2013) by Google's DeepMind.

Filter by
Sorted by
Tagged with
5
votes
2answers
94 views

Why don't people use projected Bellman error with deep neural networks?

Projected Bellman error has shown to be stable with linear function approximation. The technique is not at all new. I can only wonder why this technique is not adopted to use with non-linear function ...
1
vote
0answers
12 views

TD losses are descreasing, but also rewards are decreasing, increasing sigma

I'm using Q-learning with some extensions such as noisy linear layers, n-steps and double DQN. The training, however, isn't that successful, my rewards are descreasing over time after a steep ...
0
votes
0answers
16 views

DQN unlearns certain OpenAI-Gym environments

I solved the OpenAI-Gym MountainCar-v0 environment using dqn(using low-state-dimensional input). When I used the same code for solving CartPole-v0 environment, the network got trained in the reverse ...
3
votes
0answers
34 views

What is the difference between random and sequential sampling from the reply memory?

I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure. ...
1
vote
1answer
28 views

DQN regarding fitting off actions

While studying DQN, I rarely if ever see an explanation of what to fit off actions to. Meaning, if the highest valued action of 3 choices if the first, when training, what do we do with the other two ...
3
votes
1answer
32 views

How important is the choice of the initial state?

Is it crucial to always have the same initial (starting) state for Reinforcement Learning, for example, for Q-learning or DQN? Or it can vary?
3
votes
1answer
71 views

How is the gradient of the loss function in DQN derived?

In the original DQN paper, page 1, the loss function of the DQN is $$ L_{i}(\theta_{i}) = \mathbb{E}_{(s,a,r,s') \sim U(D)} [(r+\gamma \max_{a'} Q(s',a',\theta_{i}^{-}) - Q(s,a;\theta_{i}))^2] $$ ...
2
votes
1answer
36 views

Is a state that includes only the past n-step price records partially observable?

I'm currently working on a project to make an DQN agent that decides whether to charge or discharge an electric vehicle according to hourly changing price to sell or buy. The price pattern also varies ...
2
votes
0answers
26 views

Should importance sample weighting be compensated for by dynamically increasing learning rate?

I'm using Prioritized Experience Replay (PER) with a DDQN. To compensate for overfitting relatively high-value samples due to the non-uniform selection, I'm training with sample weights provided along ...
2
votes
1answer
32 views

How to represent players in a multi agent environment so each model can distinguish its own player

So I have 2 models trained with DQN algorithm that I want to train in a multi-agent environment to see how they react with each other. The models were trained in an environment consisting of 0's and 1'...
3
votes
1answer
37 views

Can exogenous variables be state features in reinforcement learning?

I have a question about state representation of Q-learning or DQN algorithm. I'm still a beginner of RL, so I'm not sure that is it suitable to take exogenous variables as state features. For example,...
0
votes
0answers
19 views

Suggestion on How to Push SARS to Memory Buffer of Losing RL Agent in Adversarial Learning

I am trying to implement my first RL program where there are multiple agents, rather than just one. The environment I am using is the connect four game, which is turn-based. In DQ-Learning, an agent ...
2
votes
1answer
84 views

What are the differences between the DQN variants?

There are several variants of the DQN model. For example, double DQN, duelling DQN, prioritized DQN, distributed prioritized DQN, episodic memory DQN, asynchronous n-step DQN and multiple DQN. What ...
1
vote
3answers
88 views

Reinforcement learning: How to deal with illegal actions?

I'm a beginner of RL and currently trying to make DQN agent that can act optimally in a simple situation. In the situation agent should decide at what rate to charge or discharge the electrical ...
1
vote
1answer
77 views

Deep Q Learning for Simple Game Not Effective

This is a follow-up question about one I asked earlier. The first question is here. Basically, I have a game where a paddle moves left and right to catch as much "food" as possible. Some food is good (...
0
votes
1answer
62 views

Deep Reinforcement Learning: Rewards suddenly dip down

I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using ...
4
votes
3answers
438 views

Huge action space size in Reinforcement Learning

I am working on creating a RL based AI for a certain board game. Just as a general overview of the game so that you understand what it's all about: It's a discrete turn-based game with a board of size ...
-2
votes
1answer
91 views

Problem over DQN Algorithm not converging on snake

I'm using a DQN Algorithm to play Snake. The input of the neural network is a stack of 4 images taken from the games 80x80. The output is an array of 4 values, one for every direction. The problem ...
2
votes
1answer
34 views

Will the target network, which is less trained than the normal network, output inferior estimates?

I'm having some trouble understanding some parts of the usage of target networks. I get that having the same network predict the state/action/advantage values for both the current networks can lead ...
0
votes
1answer
69 views

Is DDPG just for deterministic environments?

I want to develop an AI for continuous space. I reached to DDPG algorithm that takes actions deterministically. If DDPG takes actions deterministically, should the environment also be deterministic? ...
3
votes
1answer
52 views

DQN in stochastic environment

I'm trying to apply a DQN to a stochastic environment but i'm having trouble getting it to converge. I found some similar questions asked here, but no solutions yet. I can fairly easy get the DQN to ...
3
votes
0answers
66 views

Deep Q-Network (DQN) to learn the game 2048

I am trying to build a Deep Q-Network (DQN) agent that can learn to play the game 2048. I am orientating myself on other programs and articles that are based on the game snake and it worked well (...
1
vote
0answers
51 views

Gym dict space as keras DQN agent input

I'm trying to make an AI to play my own card game. I have an OpenAI gym for the game with Dict as an observation space. It is nested dict, so I can't easily replace it with a tuple. I want to pass ...
0
votes
0answers
50 views

Online normalization of database for DQN

I have an issue with the normalization of the database (a large time series) for my DQN. I obtained optimal results and saved the NN (5 LSTM layers) weights training on a database normalized as such: ...
2
votes
1answer
34 views

New transition priorities in Prioritized Experience Replay?

I am having a hard time converting line 6 of the prioritized experience replay algorithm from the original paper into plain English (see below): I understand that new transitions (not visited before) ...
3
votes
1answer
69 views

Experience Replay Not Always Giving Better Results

I have recently started working on a control problem using a Deep Q Network as proposed by DeepMind (https://arxiv.org/abs/1312.5602). Initially, I implemented it without Experience Replay. The ...
2
votes
1answer
68 views

Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?

In the original prioritized experience replay paper, the authors track $\gamma_t$ in every state transition tuple (see line 6 in algorithm below): Why do the authors track this at every time step? ...
3
votes
1answer
36 views

How do I represent a multi-dimensional state using a neural network?

I have a set of 15 unique playing cards from a deck of 52 playing cards. A given state is represented by the respective card values in the set of 15 cards, where the card value is a prime number ...
0
votes
0answers
40 views

DQN Algorithm not Converging for HVAC Control Task

I am trying to implement the DQN algorithm for the task of HVAC control. I have the algorithm implemented using Pytorch. I know that the HVAC simulator is working as it works for other control methods....
4
votes
2answers
125 views

Deciding on a reward per each action in a given state (Q-learning)

I looked for existing posts on Stack Exchange, which kind of answer the questions about the reward system and reward function, but not specifically what I want to ask here, which is how do you ...
2
votes
1answer
519 views

Why use semi-gradient instead of full gradient in RL problems, when using function approximation?

Semi-gradient methods work well in reinforcement learning, but what is there a reason of not using the true gradient if it can be computed? I tried it on the cart pole problem with a deep Q-network ...
0
votes
0answers
111 views

DQN Q-mean values converge negatively

I'm trying to implement my own DQN. So far I think my code is good, but my Q-values (I'm getting the mean of all the values for every episode) tends to converge near-zero but negatively. It is normal? ...
1
vote
0answers
14 views

DQN ANN input vs Linear function approximator feature vector

So when using semi-gradient td(0) you need to convert your state representation into a feature vector that represents the state and as far as I know, should not be correlated. Is the input on the ANN ...
3
votes
1answer
81 views

Which kind of prioritized experience replay should I use?

The Prioritized Experience Replay paper gives two different ways of sampling from the replay buffer. One, called "proportional prioritization", assigns each transition a priority proportional to its ...
1
vote
3answers
52 views

How to stop DQN Q function from increasing during learning?

Following the DQN algorithm with experience replay: We calculate the $loss=(Q(s,a)-(r+Q(s+1,a)))^2$. Assume I have positive but changing rewards. Meaning, $r>0$. Thus, since the rewards are ...
0
votes
1answer
44 views

Encourage Deep Q to seek short-term reward

I understand that gamma is an important factor in determining the rewards for a deep Q agent, however during testing of my network I am noticing that the agent is outputting more actions to "do ...
2
votes
0answers
33 views

Why experience reply memory in DQN instead of a RNN memory?

I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
3
votes
0answers
38 views

DQN Agent not learning anymore - what can I do to fix this?

I am trying to use Deep-Q-Learning to learn an ANN which controls a 7-DOF robotic arm. The robotic arm must avoid an obstacle and reach a target. I have implemented a number of state-of-art ...
1
vote
0answers
21 views

Deciding the rewards for different actions in Pong for a DQN agent

I am attempting to implement an agent that learns to play in the Pong environment, the environment was created in PyGame and I return the pixel data and score at each frame. I use a CNN to take a ...
3
votes
0answers
95 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
4
votes
2answers
85 views

Can DQN perform better than Double DQN?

I'm training both kind of agents against an environment but DQN performs significantly better than Double DQN. As I've saw here, Double DQN use to perform better than DQN. Am I doing something wrong ...
2
votes
1answer
22 views

Why do DQNs use linear activations on cartpole?

I've been reading a lot of tutorials on DQNs for cartpole. In many of them, they have the funnel layer of the neural net be a linear activation. Why is this? Is it just a choice made by the ...
1
vote
2answers
53 views

Can gamma be greater than 1 in a DQN?

If I have a DQN, and I care A LOT about future rewards (moreso than current rewards), can I set gamma to a number greater than 1? Like 1.1 perhaps?
1
vote
1answer
46 views

DQN Q-values are static

I am working on a DDQN with 5 LSTM layers and 3 actions as output and state space of 21 features. I am dividing the dataset into episodes of 720 timesteps, for each episode the agent acts greedily for ...
0
votes
0answers
25 views

Next step after lane detection in vehicle automation

I have an RC car with a camera, I have implemented so that i can detect lanes on my track (think like a nascar track). I want to get this car to be able to go around the track autonomous. But I am ...
1
vote
1answer
48 views

How to build a DQN agent which can be trained through interactive learning?

I am trying to create a chatbot whose dialogue policy model will be trained through reinforcement learning. Dialogue Policy is responsible for selecting the action to take based on the given state of ...
0
votes
0answers
29 views

How to limit actions based on a state [duplicate]

I'm trying to implement DQN using tf-agents for simple environment. So far I have ...
1
vote
0answers
52 views

Do we need to reset the DQN network after every episode?

I was going through this implementation of Reinforcement learning where model is being trained to manage the number of bikes at a station. Here, line 78 represents the loop over all episodes (if I ...
0
votes
1answer
39 views

OpenAI-Gym excess of actions

I'm trying to replicate the DeepMind DQN paper, and actually I'm using the OpenAI-Gym enviroment. I'm trying to get a decent score with Space Invaders (using ...
1
vote
0answers
30 views

Comparison and understanding of different version of DDQN?

There are several version of DDQN floating around. Sutton gives one that is a simple symmetric random update of the two Q functions. I think other papers (Silver paper for example) use a kind of ...