Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
3
votes
0answers
65 views

Understanding the TensorFlow implementation of the policy gradient method

I was trying to understand the implementation of a basic policy gradient (REINFORCE) method using TensorFlow. I think I got almost everything. The only thing that still bothers me is the loss function ...
1
vote
0answers
58 views

When does Monte Carlo linear function approximation converge?

In this Stanford lecture (minute 35:47 and 37:00), the professor says that Monte Carlo (MC) linear function approximation does not always converge, and she gives an example. In general, when does MC ...
2
votes
2answers
321 views

Why is DDPG an off-policy RL algorithm?

In DDPG, if there are no $\epsilon$-greedy and no action noise, is DDPG an on-policy algorithm?
4
votes
1answer
69 views

How can a single sample represent the expectation in gradient temporal difference learning?

I was reading the gradient temporal difference learning version 2(GTD2) from rich Sutton's book page-246. At some point, he expressed the whole expectation using a single sample from the environment. ...
1
vote
1answer
98 views

What does the notation $\partial \theta_{\pi}$ mean in this actor-critic update rule?

One of the steps in the actor-critic algorithm is $$\partial \theta_{\pi} \gets \partial \theta_{\pi} + \nabla_{\theta}\log\pi_{\theta} (a_i | s_i) (R - V_{\theta}(s_i))$$ For me, $\theta$ are just ...
5
votes
2answers
200 views

Why AlphaGo didn't use Deep Q-Learning?

In the previous research, in 2015, Deep Q-Learning shows its great performance on single player Atari Games. But why do AlphaGo's researchers use CNN + MCTS instead of Deep Q-Learning? is that because ...
1
vote
0answers
31 views

Why are Dueling Q Networks not used more often to approximate Q-values in reinforcement learning algorithms?

I've just learned about Dueling Network Architectures to estimate $Q$-values and am wondering why this architecture is not used more often in deep RL algorithms? DDPG and TD3 estimate the $Q$-function ...
1
vote
0answers
25 views

Do the variance and bias belong to the policy or value functions?

Recently, I read many papers on variance and bias. But I am still confused by the two notions, the variance or bias belongs to who? Policy or value? If the variance or bias is large or low, what ...
5
votes
3answers
997 views

What is the target Q-value in DQNs?

I understand that in DQNs, the loss is measured by taking the MSE of outputted Q-values and target Q-values. What does the target Q-values represent? And how is it obtained/calculated by the DQN?
5
votes
2answers
289 views

Is it possible to guide a reinforcement learning algorithm?

I have just started to study reinforcement learning and, as far as I understand, existing algorithms search for the optimal solution/policy, but do not allow the possibility for the programmer to ...
0
votes
1answer
57 views

What is the best measurement for how good an action of a reinforcement learning agent really is?

Even when we get a valuable reward signal after every single action, this immediate reward only approximates the short term goodness of the action. To consider the long term effect of an action, we ...
4
votes
1answer
127 views

Are Q values estimated from a DQN different from a duelling DQN with the same number of layers and filters?

I am confused about the Q values of a duelling deep Q network (DQN). As far as I know, duelling DQNs have 2 outputs Advantage: how good it is to be in a particular state $s$ Value: the advantage of ...
1
vote
0answers
54 views

NaNs after a while in training of PPO

My problem is that every time I am trying to train my PPO agent I get NaN values after a while. The diagnostic that I get is the following: ...
3
votes
0answers
40 views

How does normalization of the inputs work in the context of PPO?

What does the normalization of the inputs mean in the context of PPO? At each time step of an episode, I only know the values of this time step and of the previous ones, if I take track of them. This ...
2
votes
1answer
231 views

What are the most common deep reinforcement learning algorithms and models apart from DQN?

Recently, I have completed Atari Breakout (https://arxiv.org/pdf/1312.5602.pdf) with DQN. Similar to DQN, what are the most common deep reinforcement learning algorithms and models in 2020? It seems ...
2
votes
0answers
34 views

How was the DQN trained to play many games?

Some people claim that DQN was used to play many Atari games. But what actually happened? Was DQN trained only once (with some data from all games) or was it trained separately for each game? What was ...
0
votes
0answers
46 views

How should I define the loss function when using DQN to estimate the probability density?

I'm doing a Deep Q-learning project. All of my rewards are positive and there are two terminal states. One of them has a zero reward and the other has a high positive reward. The rewards are ...
1
vote
1answer
263 views

Can this be a possible deep q learning pseudocode?

I am not using replay here. Can this be a possible deep q learning pseudocode? ...
3
votes
0answers
578 views

How much time does it take to train DQN on Atari environment?

I am trying to build a DQN model for the Atari Pong game, but I am not sure whether the model is learning at all. I am using the architecture described in the paper Playing Atari with Deep ...
2
votes
1answer
145 views

Which deep reinforcement learning algorithm is appropriate for my problem?

My task is to solve an optimization problem with deep reinforcement learning. I read about several algorithms like DQN, PPO, DDPG, and A2C/A3C but use cases always seem to be problems like video games ...
5
votes
2answers
289 views

What are some online courses for deep reinforcement learning?

What are some (good) online courses for deep reinforcement learning? I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
3
votes
0answers
145 views

How to correctly implement self-play with DQN?

I have an environment where an agent faces an equal opponent, and while I've achieved OK performance implementing DQN and treating the opponent as a part of the environment, I think performance would ...
3
votes
0answers
53 views

Is this a good approach to solving Atari's “Montezuma's Revenge”?

I'm new to Reinforcement Learning. For an internship, I am currently training Atari's "Montezuma's Revenge" using a double Deep Q-Network with Hindsight Experience Replay (HER) (see also ...
4
votes
1answer
93 views

How does the optimization process in hindsight experience replay exactly work?

I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the ...
1
vote
2answers
90 views

Continuous control with DDPG: How to eliminate steady state error?

Currently I'm working on a continuous control problem using DDPG as my RL algorithm. All in all, things are working out quite well, but the algorithm does not show any tendencies to eliminate the ...
1
vote
1answer
48 views

How to let an RL Agent move the mouse?

Gday guys, I'am building a game enviroment (picture) where an agent should position the mouse on the screen (cords upper right corner) and then click to shoot a canonball. If the goal (left) is hit. ...
3
votes
1answer
64 views

Can experience replay be used for training after completing every single epoch?

The DQN implements replay memory. Based on my research, I believe the replay memory starts to get used for training once there is enough experience in the memory buffer. This means the neural network ...
2
votes
1answer
85 views

Unexpected results when comparing a greedy policy to a DQN policy

I am trying to work on a variation of the Access-Control Queuing Task problem presented in Chapter 10 of Sutton’s reinforcement learning book [1]. Specific details of my setup are as follows: I ...
2
votes
1answer
72 views

Monte Carlo updates on policy gradient with no terminal state

Consider some MDP with no terminal state. We can apply bootstrapping methods (like TD(0)) to learn in these cases no problem, but in policy gradient algorithms that have only a simple monte carlo ...
2
votes
0answers
129 views

How does adding noise to the action in DDPG help in learning?

I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.
3
votes
1answer
59 views

In the policy gradient equation, is $\pi(a_{t} | s_{t}, \theta)$ a distribution or a function?

I am learning about policy gradient methods from the Deep RL Bootcamp by Peter Abbeel and I am a bit stumbled by the math presented. In the lecture, he derives the gradient logarithm likelihood of a ...
2
votes
1answer
130 views

Should I grey-scale the coloured frames/channels to build the approximation of the state?

I'm doing reinforcement learning, and I have a visual observation that I will use to build an input state for my agent. In the DeepMind's Atari paper, they greyscale the input image before they input ...
1
vote
1answer
48 views

How to estimate the error during training in deep reinforcement learning

How do I calculate the error during the training phase for deep reinforcement learning models? Deep reinforcement learning is not supervised learning as far as I know. So how can the model know ...
4
votes
1answer
71 views

How does policy evaluation work for continuous state space model-free approaches?

How does policy evaluation work for continuous state space model-free approaches? Theoretically, a model-based approach for the discrete state and action space can be computed via dynamic programming ...
3
votes
1answer
107 views

Can recovering a reward function using IRL lead to better policies compared to reward shaping?

I am working on a research project about the different reward functions being used in the RL domain. I have read up on Inverse Reinforcement Learning (IRL) and Reward Shaping (RS). I would like to ...
2
votes
1answer
67 views

Immediate reward received in Atari game using DQN

I am trying to understand the different reward functions modelled in a reinforcement learning problem. I want to be able to know how the temporal credit assignment problem, (where the reward is ...
7
votes
1answer
140 views

Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?

While reading the DQN paper, I found that randomly selecting and learning samples reduced divergence in RL using a non-linear function approximator (e.g a neural network). So, why does Reinforcement ...
1
vote
2answers
2k views

Understanding the loss function in deep Q-learning

I am trying to understand how deep Q learning (DQN) works. To my current understanding, each $Q(s, a)$ functions is estimated to be a function of a feature vector of its state $\phi$(s) and the weight ...
3
votes
1answer
95 views

Optimal RL function approximation for TicTacToe game

I modeled the TicTacToe game as a RL problem - with an environment and an agent. At first I made an "Exact" agent - using the SARSA algorithm, I saved every unique state, and chose the best (...
1
vote
0answers
33 views

Hyperparameter optimisation over entire range or shorter range of training episodes in Deep Reinforcement Learning

I am optimising hyperparameters for my deep reinforcement learning project (using PPO2, DQN and A2C) and was wondering: Should I find the optimum hyperparameters to get maximum reward from training ...
3
votes
0answers
43 views

How to deal with nonstationary rewards in asymmetric self-play reinforcement learning?

Suppose we're training two agents to play an asymmetric game from scratch using self play (like Zerg vs. Protoss in Starcraft). During training one of the agents can become stronger (discover a good ...
1
vote
0answers
37 views

Is the training of multi-version of the same system at the same time affecting the results?

I'm using DQN to train multi-version of the same system and there is a small difference when I run them both separately. However, my result suddenly dropped in both versions if I run them both at the ...
3
votes
1answer
96 views

Purpose of using actor-critic algorithms under deterministic MDP dynamics?

One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from ...
4
votes
1answer
118 views

What could be the cause of the drop in the reward in A3C?

The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause. The problem has a sparse reward, batch size=2000, <...
3
votes
1answer
308 views

How do we compute the target value when the agent ends up in the terminal state?

I am working on a deep reinforcement learning problem. Throughout the episode, there is a small positive and negative reward for good or bad decisions. In the end, there is a huge reward for the ...
0
votes
0answers
63 views

DQN unlearns certain OpenAI-Gym environments

I solved the OpenAI-Gym MountainCar-v0 environment using dqn(using low-state-dimensional input). When I used the same code for solving CartPole-v0 environment, the network got trained in the reverse ...
2
votes
1answer
89 views

When training a DQN, how should we update the value of actions that were not taken?

Let's say that we have three actions. The highest-valued action of the three choices is the first. When training the DQN, what do we do with the other two, as we don't have a target for them, since ...
4
votes
1answer
1k views

How is the gradient of the loss function in DQN derived?

In the original DQN paper, page 1, the loss function of the DQN is $$ L_{i}(\theta_{i}) = \mathbb{E}_{(s,a,r,s') \sim U(D)} [(r+\gamma \max_{a'} Q(s',a',\theta_{i}^{-}) - Q(s,a;\theta_{i}))^2] $$ ...
2
votes
1answer
104 views

When does AlphaZero play suboptimal moves?

If AlphaZero was always playing the best moves it would just generate the same training game over and over again. So where does the randomness come from? When does it decide not to play the most ...
2
votes
0answers
53 views

Torch CNN not training

I am completely new to CNN's, and I do not quite know how to design or use them efficiently. That being said, I am attempting to build a CNN that learns to play Pac-man with reinforcement learning. I ...