Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
0
votes
1answer
54 views

What are some (deep) reinforcement learning books for beginners?

What are some books on reinforcement learning (RL) and deep RL for beginners? I'm looking for something as friendly as the head first series, that breaks down every single thing.
0
votes
0answers
27 views

How does DQN convergence work in reinforcement learning

In supervised learning we have an unbiased target value, but in reinforcement learning this isn’t the case The network predicts its own target value, now how exactly does it converge if the network ...
0
votes
0answers
21 views

How to combine two differently equally important signals into the reward function, that have different scales?

I have two signals that I want to use to model my reward. The first one is the CPU TIME: running mean from this diagram: The second one is the MAX RESIDUAL from this diagram: Since they are both ...
0
votes
0answers
11 views

Which is the best RL algo for continuous states but discrete action spaces problem

I am trying to train an AI with an environment where the states are continuous but the actions are discrete, that means I can not apply DDPG or TD3. Can someone please help to let know what should be ...
0
votes
0answers
30 views

Why do we use the Target Network for action evaluation in Double deep Q networks

Is there any specific reason as to why The target Network is used for evaluation and The online network Is used for selection, what would be the difference if both roles were switched, our online ...
1
vote
0answers
31 views

How can deep Q-learning converge if the targets may not be correct?

In deep Q-learning, $Q(s, a)$ and $Q'(s, a)$ are predicted or estimated by the neural network itself. In supervised learning, the target value is a true unbiased value. However, this isn't the case in ...
3
votes
1answer
39 views

Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning?

Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning? I could not find a clear answer on this. I would say yes it is Multi Agent Learning, as there are two Agents playing against each ...
0
votes
0answers
36 views

Having trouble understanding how Double deep Q networks work

I’ve looked at various articles and I’m still very confused, I understand the normal double Q learning about having two Action value estimates that use two different set of samples But coming to ...
-2
votes
0answers
46 views

Taking the mean as an estimate in Double Q learning

enter image description here My question is , from the images above you can see there was a normal distribution of the rewards when going left , I do not understand why they took the average of the ...
1
vote
0answers
66 views

What happens if our target network overestimates the value?

When we use DDQN, we often use the target network in case our online network overestimates a value, but this doesn't make sense to me, because What happens if our target network is the one that ...
1
vote
1answer
82 views

What exactly is the advantage of DDQN over DQN

I started looking into DDQN and apparently the difference is we use our Online network for action selection, And we use our target network for outputting the Q values, I don’t quite get how this is ...
0
votes
1answer
67 views

Do I need a large pool of training data to train a bot to play the 'pegging' game in the cribbage card game

The game of cribbage https://en.wikipedia.org/wiki/Cribbage is a two-player card game played over a series of deal with the goal to reach 121 points. The game's elements are: the discard. There are ...
1
vote
1answer
56 views

What is the difference between vanilla policy gradient and advantage actor-critic?

What is the difference between vanilla policy gradient (VPG) with a baseline as value function and advantage actor-critic (A2C)? By vanilla policy gradient I am specifically referring to spinning up's ...
0
votes
0answers
40 views

How can I build a deep reinforcement learning model that can be trained with multiple time series datasets

I built a DRL model to trade stocks in the financial market but the number of observations is relatively small and I would like to increase it by training the same model with stocks from several ...
2
votes
1answer
59 views

How does the target network in double DQNs find the maximum Q* value for each action?

I understand the fact that the neural network is used to take the states as inputs and it outputs the Q-value for state-action pairs. However, in order to compute this and update its weights, we need ...
1
vote
1answer
52 views

Why do we need target network in deep Q learning? [duplicate]

I already know deep RL, but to learn it deeply I want to know why do we need 2 networks in deep RL. What does the target network do? I now there is huge mathematics into this, but I want to know deep ...
2
votes
1answer
51 views

Why do some DQN implementations not require random exploration but instead emulate all actions?

I've found online some DQN algorithms that (in a problem with a continuous state space and few actions, let's say 2 or 3), at each time step, compute and store (in the memory used for updating) all ...
1
vote
0answers
31 views

What is the role of embeddings in a deep recurrent Q network?

When describing the model architecture for a deep recurrent q network, the authors of the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning each agent consists of a recurrent ...
0
votes
0answers
35 views

Why is my DDPG agent (implemented in TensorFlow) not learning?

I am trying to implement a Reinforcement Learning algorithm called DDPG in TensorFlow 2.x on a custom gym environment. I am new to TF. So, I started with the DDPG TF 1.x implementation from pemami4911....
2
votes
1answer
40 views

Should illegal moves be excluded from loss calculation in DQN algorithm?

I'm implementing DQN algorithm to train my agent to play a turn-based game. The action space for the game is small, but not all moves are available at all the states. Therefore, when deciding on which ...
0
votes
1answer
43 views

Why do we update the weights of the target network in deep Q learning?

I know we keep the target network constant during training to improve stability, but why exactly are we updating the weights of our target network? In particular, if we've already reached convergence, ...
0
votes
0answers
25 views

How is centralised training and decentralised execution in multi agent reinforcement learning implemented?

In the paper Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning, it is written We allow centralised training but require decentralised execution, from which follows that the policies ...
2
votes
1answer
55 views

What is the bias-variance trade-off in reinforcement learning?

I am watching DeepMind's video lecture series on reinforcement learning, and when I was watching the video of model-free RL, the instructor said the Monte Carlo methods have less bias than temporal-...
1
vote
1answer
35 views

What would happen if we sampled only one tuple from the experience replay?

The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right? What would happen if we calculate our ...
2
votes
1answer
65 views

Two DQNs in two different time scales

I have the following situation. An agent plays a game and wants to maximize the accumulated reward as usual, but it can choose its adversary. There are $n$ adversaries. In episode $e$, the agent must ...
1
vote
1answer
55 views

In Deep Q-learning, are the target update frequency and the batch training frequency related?

In a Deep Q-learning algorithm, we perform a batch training every train_freq and we update the parameters of the target network every ...
0
votes
0answers
43 views

Model Based rl and cross entropy method with nonlinear function approximators

Pseudo code for Cross entropy method according to youtube lecture 32:55 Initialize $\mu \in R^{d}, \sigma \in R^{d}$ iteration 1,2,... Collect n samples of $\theta_{i} \sim N(\mu,diag(\sigma))$ ...
0
votes
2answers
94 views

When we use a neural network to approximate the Q values, is the Q target a single value?

I have two questions When we use our network to approximate our Q values, is the Q target a single value? During backpropagation, when the weights are updated, does it automatically update the Q ...
2
votes
0answers
39 views

If the minimum Q value is decreasing and the maximum Q value increasing, is this a sign that dueling double DQN is diverging?

I'm training a dueling double DQN agent with prioritized replay buffer and notice that the min Q values are decreasing, while the max Q values are increasing. Is this a sign that it is diverging? ...
1
vote
1answer
90 views

How to implement RAM versions of Atari games

I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in ...
3
votes
0answers
77 views

How to take actions at each episode and within each step of the episode in deep Q learning?

In deep Q learning, we execute the algorithm for each episode, and for each step within an episode, we take an action and record a reward. I have a situation where my action is 2-tuple $a=(a_1,a_2)$. ...
2
votes
1answer
134 views

How to train a reinforcement learning agent from raw pixels?

How would you train a reinforcement learning agent from raw pixels? For example, if you have 3 stacked images to sense motion, then how would you pass them to neural networks to output Q-learning ...
1
vote
1answer
44 views

How to predict Q-values based on stack of frames

I decided to train deep Q-learning agent based on getting raw pixels from environment.I have one particular problem:when I input stack of frames, suppose 4 consecutive frames, if action space is 6,...
1
vote
0answers
71 views

How can the target rely on untrained parameters?

I'm trying to understand DQN. I understand where the loss function comes from. I'm just unsure about why the target function works in practice. Given the loss function $$ L_i(\theta_i) = [(y_i - Q(s,a;...
2
votes
0answers
33 views

Are the final states not being updated in this $n$-step Q-Learning algorithm?

I am reading this paper and in algorithm 3 they describe an $n$-step Q-Learning algorithm. Below is the pseudo-code. From this pseudo-code, it looks as though the final tuples that they would ...
1
vote
1answer
90 views

What is a RAM state in the gym's breakout-ram environment?

I have encountered the gym environment and decided to create AI that plays breakout. Here is the link: https://gym.openai.com/envs/Breakout-ram-v0/. The documentation says that the state is ...
1
vote
0answers
44 views

How to prevent deep Q-learning algorithms to overfit?

I have recently solved the Cartpole problem using double deep Q-learning. When I saw how the agent was doing, it used to go right every time, never left, and it did similar actions all the time. Did ...
1
vote
1answer
49 views

If deep Q-learning starts to choose only one action, is this a sign that the algorithm diverged?

I'm working on a deep q-learning model in an infinite horizon problem, with a continous state space and 3 possible actions. I'm using a neural network to approximate the action-value function. ...
4
votes
2answers
353 views

Is there any good reference for double deep Q-learning?

I am new in reinforcement learning, but I already know deep Q-learning and Q-learning. Now, I want to learn about double deep Q-learning. Do you know any good references for double deep Q-learning? ...
2
votes
1answer
49 views

How should I decay $\epsilon$ in Q-learning?

How should I decay the $\epsilon$ in Q-learning? Currently, I am decaying epsilon as follows. I initialize $\epsilon$ to be 1, then, after every episode, I multiply it by some $C$ (let it be $0.999$)...
1
vote
1answer
64 views

How and when should we update the Q-target in deep Q-learning?

I have recently watched David silver's course, and started implementing the deep Q-learning algorithm. I thought I should make a switch between the Q-target and Q-current directly (meaning, every ...
1
vote
0answers
30 views

Can you find another reason for sample inefficiency of model-free on-policy Deep Reinforcement Learning?

The following mindmap gives an overview of multiple reasons for sample inefficiency. The list is definitely not complete. Can you see another reason not mentioned so far? Some related links: ...
1
vote
1answer
62 views

Why do we also need to normalize the action's values on continuous action spaces?

I was reading here tips & tricks for training in DRL and I noticed the following: always normalize your observation space when you can, i.e., when you know the boundaries normalize your ...
1
vote
1answer
101 views

Handle non-existing states in q-learning

I am using Q-learning to solve an engineering problem. The objective is to generate a Q-table associating state to Q-values. I created a State vector ...
1
vote
1answer
39 views

Do we have two Q-learning update formulas?

I have seen two deep Q-learning formulas: $$Q\left(S_{t}, A_{t}\right) \leftarrow Q\left(S_{t}, A_{t}\right)+\alpha\left[R_{t+1}+\gamma \max _{a} Q\left(S_{t+1}, a\right)-Q\left(S_{t}, A_{t}\right)\...
3
votes
1answer
42 views

If agent chooses an action that the environment can't operate, how should I handle this situation?

I'm building a really simple experiment, letting an agent move from the bottom-left corner to the upper-right corner on a 3x3 squared paper. I plan to use DQN to do this. I'm having trouble handling ...
4
votes
1answer
55 views

How does the repetition of features across states at different time steps affect learning?

Let's say you are training a neural network in an RL setting, where the state (i.e. features/input data) can be the same for multiple successive steps (~typically around 8 steps) of an episode. For ...
2
votes
0answers
27 views

Does the concept of validation loss apply to training deep Q networks?

In deep learning, the concept of validation loss is to ensure that the model being trained is not currently overfitting the data. Is there a similar concept of overfitting in deep q learning? Given ...
1
vote
0answers
28 views

Should we start with a small batch-size and increase during training to improve sample efficiency?

Just made an interesting observation playing around with the stable-baseline's implementation of PPO and the BipedalWalker environment from OpenAI's Gym. But I believe this should be a general ...
1
vote
1answer
46 views

Is it possible to prove that the target policy is better than the behavioural policy based on learned Q values?

I have retrospective data for a sort of "behaviour policy" which I will use to train a deep q network to learn a target greedy policy. After learning the Q values for this target policy, can we make ...