Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
2
votes
1answer
30 views

Will the target network, which is less trained than the normal network, output inferior estimates?

I'm having some trouble understanding some parts of the usage of target networks. I get that having the same network predict the state/action/advantage values for both the current networks can lead ...
2
votes
1answer
40 views

If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy?

I wrote a simple implementation of Flappy Bird in Python, and now I'm trying to train an agent to play it at a reasonable skill level using TFLearn. I feed the network an input vector of size 4: ...
0
votes
0answers
30 views

Questions performance SimPLe pong for AI demo

For a demo I need to develop an AI solution to learn how to play pong. I have the following requirements: Computer needs to play against a human player. Learn while playing the game. Poor AI result ...
0
votes
1answer
49 views

Is DDPG just for deterministic environments?

I want to develop an AI for continuous space. I reached to DDPG algorithm that takes actions deterministically. If DDPG takes actions deterministically, should the environment also be deterministic? ...
1
vote
1answer
64 views

Reinforcement Learning State Definition

I am quite new to Deep Reinforcement Learning, and I'm trying to define states in a Reinforcement Learning problem. The environment consists of multiple identical elements, and each one of them is ...
1
vote
1answer
66 views

Why are we using all hyperparameters in RL?

I am new in RL and I am trying to understand why do we need all these hyperparameters. Can somebody explain me why we use them and what are the best values to use for them? total_episodes = 50000 ...
3
votes
0answers
58 views

Deep Q-Network (DQN) to learn the game 2048

I am trying to build a Deep Q-Network (DQN) agent that can learn to play the game 2048. I am orientating myself on other programs and articles that are based on the game snake and it worked well (...
1
vote
0answers
41 views

Can next state and action be same in Deep Deterministic Policy Gradient?

I am trying to apply deep deterministic policy gradient (DDPG) on a robotic application. My states consist of the joint angle positions of the robot and my actions are also its joint angle positions. ...
2
votes
1answer
44 views

Reward does not increase for a maze escaping problem with DQN

I am using deep reinforcement learning to solve a classic maze escaping task, similar to the implementation provided here, except the following three key differences: instead of using a ...
2
votes
1answer
29 views

New transition priorities in Prioritized Experience Replay?

I am having a hard time converting line 6 of the prioritized experience replay algorithm from the original paper into plain English (see below): I understand that new transitions (not visited before) ...
2
votes
1answer
68 views

Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?

In the original prioritized experience replay paper, the authors track $\gamma_t$ in every state transition tuple (see line 6 in algorithm below): Why do the authors track this at every time step? ...
1
vote
0answers
21 views

Training a reinforcement learning model with multiple images

I am tentatively trying to train a deep reinforcement learning model the maze escaping task, and each time it takes one image as the input (e.g., a different "maze"). Suppose I have about $10K$ ...
3
votes
1answer
73 views

beautify an image with reinforcement learning

I am trying to formulate and solve the following problem of image mutation. Suppose I am trying to insert an object image into a "background" image of several objects, and I will need to look for a "...
2
votes
0answers
32 views

How does the TRPO surrogate loss account for the error in the policy?

In the Trust Region Policy Optimization (TRPO) paper, on page 10, it is stated An informal overview is as follows. Our proof relies on the notion of coupling, where we jointly define the ...
1
vote
0answers
40 views

Do we need to use the experience replay buffer with the A3C algorithm?

I have skimmed through a bunch of deep learning books, but I have not yet understood whether we must use the experience replay buffer with the A3C algorithm. The approached I used is the following: ...
1
vote
0answers
15 views

What are a list of board game environments for RL practice?

Recently OpenAI removes their board game environments. (It may be possible to install an older version to get access to them, but I haven’t downgraded). Is there a list of repositories or resources ...
1
vote
0answers
62 views

Why overfitting is bad in DQN?

It is mentioned by Fu 2019 that overfitting might have a negative effect on training DQN. They showed that with either early stopping or experience replay this effect could be reduced. The first is ...
2
votes
0answers
30 views

Why experience reply memory in DQN instead of a RNN memory?

I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
3
votes
0answers
37 views

DQN Agent not learning anymore - what can I do to fix this?

I am trying to use Deep-Q-Learning to learn an ANN which controls a 7-DOF robotic arm. The robotic arm must avoid an obstacle and reach a target. I have implemented a number of state-of-art ...
3
votes
0answers
58 views

Why don't people use projected Bellman error with deep neural networks?

Projected Bellman error has shown to be stable with linear function approximation. The technique is not at all new. I can only wonder why this technique is not adopted to use with non-linear function ...
3
votes
0answers
39 views

Deep Q-Learning agent poor performing actions. Need help optimizing

I'm trying to make deep q-learning agent from https://keon.io/deep-q-learning My environment looks like this: https://imgur.com/a/OnbiCtV As you can see my agent is a circle and there is one gray ...
3
votes
0answers
84 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
9
votes
2answers
195 views

Why doesn't Q-learning converge when using function approximation?

The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions regarding the learning rate are satisfied $\sum_{t} \alpha_t(s, a) = \infty$ $...
2
votes
2answers
219 views

How large should the replay buffer be?

I'm learning DDPG algorithm by following the following link: Open AI Spinning Up document on DDPG, where it is written In order for the algorithm to have stable behavior, the replay buffer should ...
4
votes
2answers
86 views

Is reinforcement learning using shallow neural networks still deep reinforcement learning?

Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep. For example, PPO (https://arxiv.org/pdf/...
0
votes
0answers
24 views

How to limit actions based on a state [duplicate]

I'm trying to implement DQN using tf-agents for simple environment. So far I have ...
0
votes
0answers
97 views

DQN Q-mean values converge negatively

I'm trying to implement my own DQN. So far I think my code is good, but my Q-values (I'm getting the mean of all the values for every episode) tends to converge near-zero but negatively. It is normal? ...
2
votes
1answer
96 views

Regarding the output layer's activation function for continuous action space problems

I'm interested in building a (deep) RL agent for solving a continuous problem (which splits something into portions). In all examples I've seen so far, e.g., solving the continuous lunar lander, ...
4
votes
0answers
39 views

How did the OpenAI 5 for Dota concatenate units?

I am no expert in the field of AI so I apologize if this is a simple/easy question. I was trying to implement a network similar to OpenAI's for another game and I noticed that I did not fully ...
2
votes
1answer
76 views

What are the differences between the DQN variants?

There are several variants of the DQN model. For example, double DQN, duelling DQN, prioritized DQN, distributed prioritized DQN, episodic memory DQN, asynchronous n-step DQN and multiple DQN. What ...
0
votes
0answers
17 views

How to serve a deep q network using tensorflow serving?

How to Serve a Deep Q Network using Tensorflow Serving. I have built a Deep Q Network using Multilayer Perceptron. Is it possible to serve it?
1
vote
0answers
29 views

Comparison and understanding of different version of DDQN?

There are several version of DDQN floating around. Sutton gives one that is a simple symmetric random update of the two Q functions. I think other papers (Silver paper for example) use a kind of ...
3
votes
2answers
95 views

Do we have to use CNN for Deep Q Learning?

I read top articles on Google Search about Deep Q-Learning: https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8 https://skymind.ai/wiki/deep-reinforcement-...
0
votes
0answers
33 views

Extend Gym Environment for recommendation with kerasRL and some questions

I want to make a simple recommendation system based on reinforcement learning using kerasRL and OpenAI Gym. My point is that I'...
2
votes
1answer
92 views

Is there a way to train an RL agent without any environment?

Following Deep Q-learning from Demonstrations, I'd like to avoid potentially unsafe behavior during early learning by making use of supervised learning with demonstration data. However, the ...
2
votes
1answer
277 views

In DQN, updating target network every N steps or slowly update every step is better?

The use of target network is to reduce the chance of value divergence which could happen with off-policy samples trained with semi-gradient objectives. In Deep Q network, semi-gradient TD is used and ...
5
votes
2answers
119 views

Is there an alternative to the use of target network?

In the context of Deep Q Network, a target network is usually utilized. The target network is a slow changing network with a changing rate as its hyperparameter. This includes both replacement update ...
4
votes
2answers
253 views

What is the difference between DQN and AlphaGo Zero?

I have already implemented a relatively simple DQN on Pacman. Now I would like to clearly understand the difference between a DQN and the techniques used by AlphaGo zero/AlphaZero and I couldn't find ...
3
votes
2answers
240 views

My DQN is stuck and can't see where the problem is

I'm trying to replicate the DeepMind paper results, so I implemented my own DQN. I left it training for more than 4 million frames (more than 2000 episodes) on SpaceInvaders-v4 (OpenAI-Gym) and it ...
2
votes
1answer
100 views

Why does Deep Q Network outputs multiple Q values?

I am learning Deep RL following this tutorial: https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8 I understand everything but one detail: This image shows ...
0
votes
1answer
53 views

Deep Q-learning is not performing well when there are several enemies

I am playing with a deep Q-learning algorithm in my own environment. The network can perform well as long as there is only one enemy. My agent can perform the following actions: ...
1
vote
1answer
60 views

Is there a relation between the size of the neural networks and speed of convergence in deep reinforcement learning?

Is there a connection between the approximator network sizes in a RL task and the speed of convergence to an (near) optimal policy or value function? When thinking about this, I came across the ...
1
vote
1answer
47 views

How do I calculate $max_{a′}Q(s′,a′,w−)$ when it is represented as a neural network?

Consider the following loss function $$ L(\mathbf{w}) = [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})]^2 $$ where $Q(s, a, \mathbf{w^-})$ and $Q(s, a, \mathbf{w})$ are ...
0
votes
1answer
74 views

How do I update the Q values of a Deep Q Network when exploring?

I am trying to implement a Deep Q Network to play Asteroids. Unfortunately, I am not sure how to calculate the Q value exactly, if I am exploring. For example, the agent is exploring for 1 second (...
1
vote
1answer
74 views

Understanding lemma 2 of the “Trust Region Policy Optimization” paper

In the Trust Region Policy Optimization paper, in Lemma 2 of Appendix A, I did not quite understand deriving inequality (31) from (30), which is: $$\bar{A}(s) = P(a \neq \tilde{a} | s) \mathbb{E}_{(a,...
6
votes
1answer
4k views

How does LSTM in deep reinforcement learning differ from experience replay?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the author processed the Atari game frames with an LSTM layer at the end. My questions are: How does this method differ from the ...
5
votes
2answers
458 views

TRPO/PPO importance sampling term in loss function

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients ...
3
votes
1answer
38 views

Maximizing or Minimizing in Trust Region Policy Optimization?

I happened to discover that the v1 (19 Feb 2015) and the v5 (20 Apr 2017) versions of TRPO papers have two different conclusions. The Equation (15) in v1 is $\min_\theta$ while the Equation (14) in v2 ...
6
votes
2answers
330 views

What is experience replay in laymen's terms?

I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...
1
vote
1answer
490 views

Why use semi-gradient instead of full gradient in RL problems, when using function approximation?

Semi-gradient methods work well in reinforcement learning, but what is there a reason of not using the true gradient if it can be computed? I tried it on the cart pole problem with a deep Q-network ...