Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
0
votes
0answers
15 views

How should I define the loss function when using DQN to estimate the probability density?

I'm doing a Deep Q-learning project. All of my rewards are positive and there are two terminal states. One of them has a zero reward and the other has a high positive reward. The rewards are ...
1
vote
1answer
30 views

Can this be a possible deep q learning pseudocode?

I am not using replay here. Can this be a possible deep q learning pseudocode? ...
2
votes
0answers
17 views

How much time does it take to train DQN on Atari environment?

I am trying to build a DQN model for the Atari Pong game, but I am not sure whether the model is learning at all. I am using the architecture described in the paper Playing Atari with Deep ...
2
votes
1answer
71 views

Which deep reinforcement learning algorithm is appropriate for my problem?

My task is to solve an optimization problem with deep reinforcement learning. I read about several algorithms like DQN, PPO, DDPG, and A2C/A3C but use cases always seem to be problems like video games ...
2
votes
2answers
74 views

What are some online courses for deep reinforcement learning?

What are some (good) online courses for deep reinforcement learning? I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
1
vote
0answers
36 views

How to correctly implement self-play with DQN?

I have an environment where an agent faces an equal opponent, and while I've achieved OK performance implementing DQN and treating the opponent as a part of the environment, I think performance would ...
1
vote
0answers
15 views

Continuous control with DDPG: How to eliminate steady state error?

Currently I'm working on a continuous control problem using DDPG as my RL algorithm. All in all, things are working out quite well, but the algorithm does not show any tendencies to eliminate the ...
1
vote
1answer
30 views

How to let an RL Agent move the mouse?

Gday guys, I'am building a game enviroment (picture) where an agent should position the mouse on the screen (cords upper right corner) and then click to shoot a canonball. If the goal (left) is hit. ...
3
votes
1answer
35 views

Can experience replay be used for training after completing every single epoch?

The DQN implements replay memory. Based on my research, I believe the replay memory starts to get used for training once there is enough experience in the memory buffer. This means the neural network ...
2
votes
1answer
61 views

Unexpected results when comparing a greedy policy to a DQN policy

I am trying to work on a variation of the Access-Control Queuing Task problem presented in Chapter 10 of Sutton’s reinforcement learning book [1]. Specific details of my setup are as follows: I ...
2
votes
1answer
53 views

Monte Carlo updates on policy gradient with no terminal state

Consider some MDP with no terminal state. We can apply bootstrapping methods (like TD(0)) to learn in these cases no problem, but in policy gradient algorithms that have only a simple monte carlo ...
2
votes
0answers
36 views

How does adding noise to the action in DDPG help in learning?

I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.
3
votes
1answer
44 views

In the policy gradient equation, is $\pi(a_{t} | s_{t}, \theta)$ a distribution or a function?

I am learning about policy gradient methods from the Deep RL Bootcamp by Peter Abbeel and I am a bit stumbled by the math presented. In the lecture, he derives the gradient logarithm likelihood of a ...
1
vote
1answer
35 views

How to estimate the error during training in deep reinforcement learning

How do I calculate the error during the training phase for deep reinforcement learning models? Deep reinforcement learning is not supervised learning as far as I know. So how can the model know ...
2
votes
0answers
12 views

How does policy evaluation work for continuous state space model-free approaches?

How does policy evaluation work for continuous state space model-free approaches? Theoretical model-based approach for the discrete state and action space can be computed via dynamic programming and ...
2
votes
0answers
32 views

Policy $\pi$ obtained from inverse reinforcement learning vs reward shaping

I am working on a research project about the different reward functions being used in RL domain. I have read up on Inverse Reinforcement Learning (IRL) and Reward Shaping (RS). I would like to clarify ...
1
vote
1answer
41 views

Immediate reward received in Atari game using DQN

I am trying to understand the different reward functions modelled in a reinforcement learning problem. I want to be able to know how the temporal credit assignment problem, (where the reward is ...
0
votes
0answers
14 views

How to pass observation from CartPole-v0 to neural network using tensorflow

I am trying to learn about RL by implementing DQN with tensorflow. However, I am really stuck with tensorflow.. I just don't understand it. I think I have found the core of what I understand - I dont ...
3
votes
0answers
63 views

Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?

While reading the DQN paper, I found that randomly selecting and learning samples reduced divergence in RL using a non-linear function approximator (e.g a neural network). So why does Reinforcement ...
1
vote
1answer
66 views

Understanding the loss function in deep Q-learning

I am trying to understand how deep Q learning (DQN) works. To my current understanding, each $Q(s, a)$ functions is estimated to be a function of a feature vector of its state $\phi$(s) and the weight ...
3
votes
1answer
40 views

Optimal RL function approximation for TicTacToe game

I modeled the TicTacToe game as a RL problem - with an environment and an agent. At first I made an "Exact" agent - using the SARSA algorithm, I saved every unique state, and chose the best (...
1
vote
0answers
20 views

Hyperparameter optimisation over entire range or shorter range of training episodes in Deep Reinforcement Learning

I am optimising hyperparameters for my deep reinforcement learning project (using PPO2, DQN and A2C) and was wondering: Should I find the optimum hyperparameters to get maximum reward from training ...
3
votes
0answers
33 views

How to deal with nonstationary rewards in asymmetric self-play reinforcement learning?

Suppose we're training two agents to play an asymmetric game from scratch using self play (like Zerg vs. Protoss in Starcraft). During training one of the agents can become stronger (discover a good ...
1
vote
0answers
32 views

Is the training of multi-version of the same system at the same time affecting the results?

I'm using DQN to train multi-version of the same system and there is a small difference when I run them both separately. However, my result suddenly dropped in both versions if I run them both at the ...
3
votes
1answer
47 views

Purpose of using actor-critic algorithms under deterministic MDP dynamics?

One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from ...
3
votes
1answer
45 views

What could be the cause of the drop in the reward in A3C?

The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause. The problem has a sparse reward, batch size=2000, <...
2
votes
1answer
51 views

Doubt in Deep-Q learning with sparse rewards

I am working on a deep reinforcement learning problem, when I got stuck at the following questions. They are rather general and not specific to my specific problem. The solution uses a sparse reward ...
0
votes
0answers
34 views

DQN unlearns certain OpenAI-Gym environments

I solved the OpenAI-Gym MountainCar-v0 environment using dqn(using low-state-dimensional input). When I used the same code for solving CartPole-v0 environment, the network got trained in the reverse ...
0
votes
0answers
22 views

Invalid moves in Deep Reinforcement Learning for games [duplicate]

I've been working on a bot for a game involving dice throws and chance. The architecture involved is similar to AlphaZero in the that it has Convolutions and MCTS. According to the current state ...
3
votes
1answer
173 views

How is the gradient of the loss function in DQN derived?

In the original DQN paper, page 1, the loss function of the DQN is $$ L_{i}(\theta_{i}) = \mathbb{E}_{(s,a,r,s') \sim U(D)} [(r+\gamma \max_{a'} Q(s',a',\theta_{i}^{-}) - Q(s,a;\theta_{i}))^2] $$ ...
1
vote
1answer
57 views

When does AlphaZero play suboptimal moves?

If AlphaZero was always playing the best moves it would just generate the same training game over and over again. So where does the randomness come from? When does it decide not to play the most ...
2
votes
0answers
50 views

Torch CNN not training

I am completely new to CNN's, and I do not quite know how to design or use them efficiently. That being said, I am attempting to build a CNN that learns to play Pac-man with reinforcement learning. I ...
2
votes
1answer
36 views

Will the target network, which is less trained than the normal network, output inferior estimates?

I'm having some trouble understanding some parts of the usage of target networks. I get that having the same network predict the state/action/advantage values for both the current networks can lead ...
2
votes
1answer
41 views

If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy?

I wrote a simple implementation of Flappy Bird in Python, and now I'm trying to train an agent to play it at a reasonable skill level using TFLearn. I feed the network an input vector of size 4: ...
0
votes
0answers
34 views

Questions performance SimPLe pong for AI demo

For a demo I need to develop an AI solution to learn how to play pong. I have the following requirements: Computer needs to play against a human player. Learn while playing the game. Poor AI result ...
0
votes
1answer
108 views

Is DDPG just for deterministic environments?

I want to develop an AI for continuous space. I reached to DDPG algorithm that takes actions deterministically. If DDPG takes actions deterministically, should the environment also be deterministic? ...
1
vote
1answer
138 views

Reinforcement Learning State Definition

I am quite new to Deep Reinforcement Learning, and I'm trying to define states in a Reinforcement Learning problem. The environment consists of multiple identical elements, and each one of them is ...
1
vote
1answer
114 views

Why are we using all hyperparameters in RL?

I am new in RL and I am trying to understand why do we need all these hyperparameters. Can somebody explain me why we use them and what are the best values to use for them? total_episodes = 50000 ...
3
votes
0answers
122 views

Deep Q-Network (DQN) to learn the game 2048

I am trying to build a Deep Q-Network (DQN) agent that can learn to play the game 2048. I am orientating myself on other programs and articles that are based on the game snake and it worked well (...
1
vote
0answers
45 views

Can next state and action be same in Deep Deterministic Policy Gradient?

I am trying to apply deep deterministic policy gradient (DDPG) on a robotic application. My states consist of the joint angle positions of the robot and my actions are also its joint angle positions. ...
2
votes
1answer
48 views

Reward does not increase for a maze escaping problem with DQN

I am using deep reinforcement learning to solve a classic maze escaping task, similar to the implementation provided here, except the following three key differences: instead of using a ...
2
votes
1answer
51 views

New transition priorities in Prioritized Experience Replay?

I am having a hard time converting line 6 of the prioritized experience replay algorithm from the original paper into plain English (see below): I understand that new transitions (not visited before) ...
4
votes
0answers
188 views

What could be causing the drastic performance drop of the DQN model on the Pong environment?

I am running a basic DQN (Deep Q-Network) on the Pong environment. Not a CNN, just a 3 layer linear neural net with ReLUs. It seems to work for the most part, but at some point, my model suffers from ...
3
votes
1answer
71 views

Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?

In the original prioritized experience replay paper, the authors track $\gamma_t$ in every state transition tuple (see line 6 in algorithm below): Why do the authors track this at every time step? ...
2
votes
0answers
24 views

Training a reinforcement learning model with multiple images

I am tentatively trying to train a deep reinforcement learning model the maze escaping task, and each time it takes one image as the input (e.g., a different "maze"). Suppose I have about $10K$ ...
3
votes
1answer
84 views

beautify an image with reinforcement learning

I am trying to formulate and solve the following problem of image mutation. Suppose I am trying to insert an object image into a "background" image of several objects, and I will need to look for a "...
2
votes
0answers
36 views

How does the TRPO surrogate loss account for the error in the policy?

In the Trust Region Policy Optimization (TRPO) paper, on page 10, it is stated An informal overview is as follows. Our proof relies on the notion of coupling, where we jointly define the ...
1
vote
0answers
131 views

Do we need to use the experience replay buffer with the A3C algorithm?

I have skimmed through a bunch of deep learning books, but I have not yet understood whether we must use the experience replay buffer with the A3C algorithm. The approached I used is the following: ...
1
vote
0answers
25 views

What are a list of board game environments for RL practice?

Recently OpenAI removes their board game environments. (It may be possible to install an older version to get access to them, but I haven’t downgraded). Is there a list of repositories or resources ...
2
votes
0answers
170 views

Why overfitting is bad in DQN?

It is mentioned by Fu 2019 that overfitting might have a negative effect on training DQN. They showed that with either early stopping or experience replay this effect could be reduced. The first is ...