Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
1
vote
0answers
15 views

What is the role of embeddings in a deep recurrent Q network?

When describing the model architecture for a deep recurrent q network, the authors of the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning each agent consists of a recurrent ...
2
votes
1answer
52 views

can recovering a reward function using IRL lead to better policies compared to reward shaping?

I am working on a research project about the different reward functions being used in the RL domain. I have read up on Inverse Reinforcement Learning (IRL) and Reward Shaping (RS). I would like to ...
0
votes
0answers
21 views

Why is my DDPG agent (implemented in TensorFlow) not learning?

I am trying to implement a Reinforcement Learning algorithm called DDPG in TensorFlow 2.x on a custom gym environment. I am new to TF. So, I started with the DDPG TF 1.x implementation from pemami4911....
6
votes
2answers
263 views

Is reinforcement learning using shallow neural networks still deep reinforcement learning?

Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep. For example, PPO is often considered a ...
3
votes
2answers
134 views

What are some online courses for deep reinforcement learning?

What are some (good) online courses for deep reinforcement learning? I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
2
votes
1answer
28 views

Should illegal moves be excluded from loss calculation in DQN algorithm?

I'm implementing DQN algorithm to train my agent to play a turn-based game. The action space for the game is small, but not all moves are available at all the states. Therefore, when deciding on which ...
0
votes
1answer
32 views

Why do we update the weights of the target network in deep Q learning?

I know we keep the target network constant during training to improve stability, but why exactly are we updating the weights of our target network? In particular, if we've already reached convergence, ...
0
votes
0answers
15 views

How is centralised training and decentralised execution in multi agent reinforcement learning implemented?

In the paper Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning, it is written We allow centralised training but require decentralised execution, from which follows that the policies ...
2
votes
1answer
37 views

What is the bias-variance trade-off in reinforcement learning?

I am watching DeepMind's video lecture series on reinforcement learning, and when I was watching the video of model-free RL, the instructor said the Monte Carlo methods have less bias than temporal-...
1
vote
1answer
25 views

What would happen if we sampled only one tuple from the experience replay?

The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right? What would happen if we calculate our ...
1
vote
2answers
147 views

Understanding the loss function in deep Q-learning

I am trying to understand how deep Q learning (DQN) works. To my current understanding, each $Q(s, a)$ functions is estimated to be a function of a feature vector of its state $\phi$(s) and the weight ...
2
votes
1answer
54 views

Two DQNs in two different time scales

I have the following situation. An agent plays a game and wants to maximize the accumulated reward as usual, but it can choose its adversary. There are $n$ adversaries. In episode $e$, the agent must ...
1
vote
1answer
41 views

In Deep Q-learning, are the target update frequency and the batch training frequency related?

In a Deep Q-learning algorithm, we perform a batch training every train_freq and we update the parameters of the target network every ...
0
votes
0answers
33 views

Model Based rl and cross entropy method with nonlinear function approximators

Pseudo code for Cross entropy method according to youtube lecture 32:55 Initialize $\mu \in R^{d}, \sigma \in R^{d}$ iteration 1,2,... Collect n samples of $\theta_{i} \sim N(\mu,diag(\sigma))$ ...
1
vote
1answer
38 views

Is it possible to prove that the target policy is better than the behavioural policy based on learned Q values?

I have retrospective data for a sort of "behaviour policy" which I will use to train a deep q network to learn a target greedy policy. After learning the Q values for this target policy, can we make ...
0
votes
0answers
5 views

Tensorboard not reflecting Logged Data [migrated]

I am new to using tensorboard and I wanted to get my own graph on the tensorboard, outside of training. I am working on Google Colabs and the code that I wrote is, ...
0
votes
2answers
84 views

When we use a neural network to approximate the Q values, is the Q target a single value?

I have two questions When we use our network to approximate our Q values, is the Q target a single value? During backpropagation, when the weights are updated, does it automatically update the Q ...
4
votes
1answer
52 views

How does policy evaluation work for continuous state space model-free approaches?

How does policy evaluation work for continuous state space model-free approaches? Theoretically, a model-based approach for the discrete state and action space can be computed via dynamic programming ...
1
vote
1answer
73 views

How to implement RAM versions of Atari games

I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in ...
14
votes
3answers
838 views

Why doesn't Q-learning converge when using function approximation?

The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions (the Robbins-Monro conditions) regarding the learning rate are satisfied $\...
2
votes
0answers
29 views

If the minimum Q value is decreasing and the maximum Q value increasing, is this a sign that dueling double DQN is diverging?

I'm training a dueling double DQN agent with prioritized replay buffer and notice that the min Q values are decreasing, while the max Q values are increasing. Is this a sign that it is diverging? ...
4
votes
1answer
48 views

How can a single sample represent the expectation in gradient temporal difference learning?

I was reading the gradient temporal difference learning version 2(GTD2) from rich Sutton's book page-246. At some point, he expressed the whole expectation using a single sample from the environment. ...
2
votes
1answer
119 views

How to train a reinforcement learning agent from raw pixels?

How would you train a reinforcement learning agent from raw pixels? For example, if you have 3 stacked images to sense motion, then how would you pass them to neural networks to output Q-learning ...
3
votes
0answers
68 views

How to take actions at each episode and within each step of the episode in deep Q learning?

In deep Q learning, we execute the algorithm for each episode, and for each step within an episode, we take an action and record a reward. I have a situation where my action is 2-tuple $a=(a_1,a_2)$. ...
1
vote
1answer
41 views

Why do we also need to normalize the action's values on continuous action spaces?

I was reading here tips & tricks for training in DRL and I noticed the following: always normalize your observation space when you can, i.e., when you know the boundaries normalize your ...
1
vote
1answer
32 views

How to predict Q-values based on stack of frames

I decided to train deep Q-learning agent based on getting raw pixels from environment.I have one particular problem:when I input stack of frames, suppose 4 consecutive frames, if action space is 6,...
4
votes
1answer
1k views

Can TD($\lambda$) be used with deep reinforcement learning?

TD lambda is a way to interpolate between TD(0) - bootstrapping over a single step, and, TD(max), bootstrapping over the entire episode length, or, Monte Carlo. Reading the link above, I see that an ...
1
vote
0answers
61 views

How can the target rely on untrained parameters?

I'm trying to understand DQN. I understand where the loss function comes from. I'm just unsure about why the target function works in practice. Given the loss function $$ L_i(\theta_i) = [(y_i - Q(s,a;...
1
vote
2answers
50 views

Continuous control with DDPG: How to eliminate steady state error?

Currently I'm working on a continuous control problem using DDPG as my RL algorithm. All in all, things are working out quite well, but the algorithm does not show any tendencies to eliminate the ...
2
votes
0answers
24 views

Are the final states not being updated in this $n$-step Q-Learning algorithm?

I am reading this paper and in algorithm 3 they describe an $n$-step Q-Learning algorithm. Below is the pseudo-code. From this pseudo-code, it looks as though the final tuples that they would ...
1
vote
1answer
77 views

What is a RAM state in the gym's breakout-ram environment?

I have encountered the gym environment and decided to create AI that plays breakout. Here is the link: https://gym.openai.com/envs/Breakout-ram-v0/. The documentation says that the state is ...
1
vote
1answer
94 views

Handle non-existing states in q-learning

I am using Q-learning to solve an engineering problem. The objective is to generate a Q-table associating state to Q-values. I created a State vector ...
1
vote
0answers
35 views

How to prevent deep Q-learning algorithms to overfit?

I have recently solved the Cartpole problem using double deep Q-learning. When I saw how the agent was doing, it used to go right every time, never left, and it did similar actions all the time. Did ...
1
vote
1answer
36 views

If deep Q-learning starts to choose only one action, is this a sign that the algorithm diverged?

I'm working on a deep q-learning model in an infinite horizon problem, with a continous state space and 3 possible actions. I'm using a neural network to approximate the action-value function. ...
4
votes
2answers
331 views

Is there any good reference for double deep Q-learning?

I am new in reinforcement learning, but I already know deep Q-learning and Q-learning. Now, I want to learn about double deep Q-learning. Do you know any good references for double deep Q-learning? ...
2
votes
1answer
40 views

How should I decay $\epsilon$ in Q-learning?

How should I decay the $\epsilon$ in Q-learning? Currently, I am decaying epsilon as follows. I initialize $\epsilon$ to be 1, then, after every episode, I multiply it by some $C$ (let it be $0.999$)...
1
vote
1answer
49 views

How and when should we update the Q-target in deep Q-learning?

I have recently watched David silver's course, and started implementing the deep Q-learning algorithm. I thought I should make a switch between the Q-target and Q-current directly (meaning, every ...
0
votes
0answers
19 views

Can you find another reason for sample inefficiency of model-free on-policy Deep Reinforcement Learning?

The following mindmap gives an overview of multiple reasons for sample inefficiency. The list is definitely not complete. Can you see another reason not mentioned so far? Some related links: ...
1
vote
1answer
25 views

Do we have two Q-learning update formulas?

I have seen two deep Q-learning formulas: $$Q\left(S_{t}, A_{t}\right) \leftarrow Q\left(S_{t}, A_{t}\right)+\alpha\left[R_{t+1}+\gamma \max _{a} Q\left(S_{t+1}, a\right)-Q\left(S_{t}, A_{t}\right)\...
4
votes
1answer
46 views

How does the repetition of features across states at different time steps affect learning?

Let's say you are training a neural network in an RL setting, where the state (i.e. features/input data) can be the same for multiple successive steps (~typically around 8 steps) of an episode. For ...
3
votes
1answer
81 views

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

Twin Delayed Deep Deterministic (TD3) policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained ...
3
votes
1answer
33 views

If agent chooses an action that the environment can't operate, how should I handle this situation?

I'm building a really simple experiment, letting an agent move from the bottom-left corner to the upper-right corner on a 3x3 squared paper. I plan to use DQN to do this. I'm having trouble handling ...
2
votes
0answers
19 views

Does the concept of validation loss apply to training deep Q networks?

In deep learning, the concept of validation loss is to ensure that the model being trained is not currently overfitting the data. Is there a similar concept of overfitting in deep q learning? Given ...
1
vote
0answers
21 views

Should we start with a small batch-size and increase during training to improve sample efficiency?

Just made an interesting observation playing around with the stable-baseline's implementation of PPO and the BipedalWalker environment from OpenAI's Gym. But I believe this should be a general ...
2
votes
2answers
166 views

What should the target be when the neural network outputs multiple Q values in deep Q-learning?

I have some gaps in my understanding regarding the performing of the gradient descent in Deep - Q networks. The original deep q network for Atari performs a gradient descent step to minimise $y_j - Q(...
1
vote
1answer
33 views

Can we combine Off-Policy with On-Policy Algorithms?

On-Policy Algorithms like PPO directly maximize the performance objective or an approximation of it. They tend to be quite stable and reliable but are often sample inefficient. Off-Policy Algorithms ...
1
vote
1answer
48 views

Why can't DQN be used for self-driving cars?

Why can't DQN be used for self-driving cars? Why can't DQN and similar RL algorithms be used for self-driving cars? The reason why I am curious is that it successfully plays go and other multistate ...
1
vote
0answers
20 views

Why is it hard to prove the convergence of the deep Q-learning algorithm?

Why is it hard to prove the convergence of the DQN algorithm? We know that the tabular Q-learning algorithm converges to the optimal Q-values, and with a linear approximator convergence is proved. ...
1
vote
0answers
24 views

State-of-the-art algorithms not working on a custom RL environment

I'm trying to train a RL agent on a custom, highly stochastic environment (MDP). In order to do so I'm using existing implementations of state-of-the-art RL algorithms as provided by Stable Baselines. ...
2
votes
1answer
103 views

Which deep reinforcement learning algorithm is appropriate for my problem?

My task is to solve an optimization problem with deep reinforcement learning. I read about several algorithms like DQN, PPO, DDPG, and A2C/A3C but use cases always seem to be problems like video games ...