Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
0
votes
0answers
19 views

Would a model-based RL algorithm perform better that a model-free one, in the air traffic control sequencing simulation field? [closed]

I am learning RL now. When I start to run my algorithm (PPO) for air traffic control sequencing, it gets trapped into local optimality. Will it be better when I choose to use a model-based RL method, ...
2
votes
0answers
23 views

Why is tree search/planning used in reinforcement learning

We know that in Alphago zero, MCTS is used along with policy networks. Some sources say MCTS (or planning in general) increases the sample efficiency. Assumed the transition model is known and the ...
2
votes
2answers
43 views

How should I define the reward function to solve the Wumpus game with deep Q-learning?

I'm writing a DQN agent for the Wumpus game. Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...
2
votes
0answers
35 views

How to model this problem to solve using DQN?

I have a scheduling problem as follows. At each time $t=1,2,\ldots,T$, a set of $n$ jobs arrives where job $j$ has a cost $c_{j,t}$ and a budget $b_{j,t}$ which are revealed to the decision maker for ...
1
vote
0answers
38 views

How to design an observation(state) space for a simple `Rock-Paper-Scissor` game?

For weeks I've been working with this toy game of Rock-Paper-Scissor. I want to use a PPO agent learn to beat a computer ...
0
votes
1answer
21 views

Why would the reward of A3C with LSTM suddenly drop off after many episodes?

I am training an A3C with stacked LSTM. During initial training, my model was giving descent +ve reward. However, after many episodes, its reward just goes to zero and is continuing for a long time. ...
0
votes
0answers
19 views

Reward and loss follow the same shape in DQN

If the accumulated reward increases, the loss increases and vice versa. This is a strange behaviour. See the figure below for an example. What is the possibility of having this behaviour in DQN? I ...
0
votes
0answers
37 views

Does Multi-Agent Deep Deterministic Policy Gradient also work with discrete action spaces?

I would like to ask if Multi-Agent Deep Deterministic Policy Gradient (MADDPG) works fine with discrete action space. DDPG works only with continuous action space, but I have read that MADDPG can also ...
0
votes
0answers
19 views

DDPG with Hindsight Experience Replay not converging properly

I am trying to train kuka arm (KukaGymEnv) on DDPG augmented with Hindsight Experience Replay (HER). The hyperparameters are as given : Actor and Critic are simple neural networks with two hidden ...
1
vote
0answers
20 views

Is it feasible to train a DQN with thousands of input ports?

I designed a DQN architecture for some problem. The problem has a parameter $m$ as the number of clients. In my situation, $m$ is large, $m\in\{100,200,\ldots,1000\}$. For this situation, the number ...
3
votes
1answer
57 views

How can I fix jerky movement in a continuous action space

I am training an agent to do object avoidance. The agent has control over its steering angle and its speed. The steering angle and speed are normalized in a $[−1,1]$ range, where the sign encodes ...
2
votes
1answer
46 views

How we are calculating average reward ($r(\pi)$) if the policy changes over time?

In the average reward setting the quality of a policy is defined as: $$ r(\pi) = \lim_{h\to\infty}\frac{1}{h} \sum_{j=1}^{h}E[R_j] $$ When we reach the steady state distribution we can write the above ...
1
vote
1answer
39 views

What is the optimal exploration-exploitation trade-off in Q*bert?

I am training an RL agent with Deep Q-learning + Experience Replay on the Q*bert Atari environment. After 400,000 frames, my agent appears to have learned strategic information about the game, but ...
3
votes
1answer
36 views

Is there a logical method of deducing an optimal batch size when training a Deep Q-learning agent with experience replay?

I am training an RL agent using Deep-Q learning with experience replay. At each frame, I am currently sampling 32 random transitions from a queue which stores a maximum of 20000 and training as ...
1
vote
0answers
44 views

Why scaling reward drastically affects performance?

I have devised an gridworld-like environment where a RL agent is tasked to cover all the blank squares by passing through them. Possible actions are up, down, left, right. The reward scheme is the ...
1
vote
1answer
52 views

What are the variables that need to be saved and loaded, so that a DQN model starts where it left off?

TensorFlow allows users to save the weights and the model architecture, however, that will be insufficient unless the values of certain other variables are also stored. For instance, in DQN, if $\...
1
vote
1answer
46 views

How is exponential moving average computed in deep Q networks?

In normal Q-learning, the update rule is an implementation of the exponential moving average, which then converges to the optimal true Q values. However, looking at DQN, how exactly is the exponential ...
4
votes
1answer
106 views

How does the Ornstein-Uhlenbeck process work, and how it is used in DDPG?

In section 3 of the paper Continuous control with deep reinforcement learning, the authors write As detailed in the supplementary materials we used an Ornstein-Uhlenbeck process (Uhlenbeck & ...
1
vote
1answer
30 views

Why are Target Networks used in Deep Q-Learning as opposed to the Expected Value equation?

I understand we use a target network because it helps resolve issues regarding stability, however, that's not what I'm here to ask. What I would like to understand is why a target network is used as a ...
0
votes
0answers
25 views

Select training data for episodic reinforcement learning (stock trading agent)

I playing around with a stock trading agent trained via (deep) reinforcement learning, including memory replay. The agent is trained for 1000 episodes, where each episodes consists of 180 timesteps (e....
1
vote
0answers
21 views

Trying to proof off policy TD Learning formula

I was reading the book "Introduction to Reinforcement Learning" by Richard Sutton In section 7.3 he write the formula for n-step off-policy TD as:. $$V(S_t) = V(S_{t-1}) + \alpha \rho_{t:t+n-...
0
votes
2answers
108 views

What are some (deep) reinforcement learning books for beginners? [duplicate]

What are some books on reinforcement learning (RL) and deep RL for beginners? I'm looking for something as friendly as the head first series, that breaks down every single thing.
0
votes
0answers
34 views

How does DQN convergence work in reinforcement learning

In supervised learning we have an unbiased target value, but in reinforcement learning this isn’t the case The network predicts its own target value, now how exactly does it converge if the network ...
0
votes
0answers
28 views

How to combine two differently equally important signals into the reward function, that have different scales?

I have two signals that I want to use to model my reward. The first one is the CPU TIME: running mean from this diagram: The second one is the MAX RESIDUAL from this diagram: Since they are both ...
0
votes
0answers
18 views

Which is the best RL algo for continuous states but discrete action spaces problem

I am trying to train an AI with an environment where the states are continuous but the actions are discrete, that means I can not apply DDPG or TD3. Can someone please help to let know what should be ...
0
votes
0answers
31 views

Why do we use the Target Network for action evaluation in Double deep Q networks

Is there any specific reason as to why The target Network is used for evaluation and The online network Is used for selection, what would be the difference if both roles were switched, our online ...
1
vote
0answers
32 views

How can deep Q-learning converge if the targets may not be correct?

In deep Q-learning, $Q(s, a)$ and $Q'(s, a)$ are predicted or estimated by the neural network itself. In supervised learning, the target value is a true unbiased value. However, this isn't the case in ...
3
votes
1answer
48 views

Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning?

Can AlphaZero considered as Multi-Agent Deep Reinforcement Learning? I could not find a clear answer on this. I would say yes it is Multi Agent Learning, as there are two Agents playing against each ...
0
votes
0answers
38 views

Having trouble understanding how Double deep Q networks work

I’ve looked at various articles and I’m still very confused, I understand the normal double Q learning about having two Action value estimates that use two different set of samples But coming to ...
1
vote
0answers
69 views

What happens if our target network overestimates the value?

When we use DDQN, we often use the target network in case our online network overestimates a value, but this doesn't make sense to me, because What happens if our target network is the one that ...
1
vote
1answer
96 views

What exactly is the advantage of DDQN over DQN

I started looking into DDQN and apparently the difference is we use our Online network for action selection, And we use our target network for outputting the Q values, I don’t quite get how this is ...
0
votes
1answer
72 views

Do I need a large pool of training data to train a bot to play the 'pegging' game in the cribbage card game

The game of cribbage https://en.wikipedia.org/wiki/Cribbage is a two-player card game played over a series of deal with the goal to reach 121 points. The game's elements are: the discard. There are ...
2
votes
1answer
95 views

What is the difference between vanilla policy gradient and advantage actor-critic?

What is the difference between vanilla policy gradient (VPG) with a baseline as value function and advantage actor-critic (A2C)? By vanilla policy gradient I am specifically referring to spinning up's ...
0
votes
0answers
41 views

How can I build a deep reinforcement learning model that can be trained with multiple time series datasets

I built a DRL model to trade stocks in the financial market but the number of observations is relatively small and I would like to increase it by training the same model with stocks from several ...
2
votes
1answer
76 views

How does the target network in double DQNs find the maximum Q* value for each action?

I understand the fact that the neural network is used to take the states as inputs and it outputs the Q-value for state-action pairs. However, in order to compute this and update its weights, we need ...
1
vote
1answer
63 views

Why do we need target network in deep Q learning? [duplicate]

I already know deep RL, but to learn it deeply I want to know why do we need 2 networks in deep RL. What does the target network do? I now there is huge mathematics into this, but I want to know deep ...
2
votes
1answer
52 views

Why do some DQN implementations not require random exploration but instead emulate all actions?

I've found online some DQN algorithms that (in a problem with a continuous state space and few actions, let's say 2 or 3), at each time step, compute and store (in the memory used for updating) all ...
1
vote
0answers
31 views

What is the role of embeddings in a deep recurrent Q network?

When describing the model architecture for a deep recurrent q network, the authors of the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning each agent consists of a recurrent ...
0
votes
0answers
46 views

Why is my DDPG agent (implemented in TensorFlow) not learning?

I am trying to implement a Reinforcement Learning algorithm called DDPG in TensorFlow 2.x on a custom gym environment. I am new to TF. So, I started with the DDPG TF 1.x implementation from pemami4911....
2
votes
1answer
44 views

Should illegal moves be excluded from loss calculation in DQN algorithm?

I'm implementing DQN algorithm to train my agent to play a turn-based game. The action space for the game is small, but not all moves are available at all the states. Therefore, when deciding on which ...
0
votes
1answer
44 views

Why do we update the weights of the target network in deep Q learning?

I know we keep the target network constant during training to improve stability, but why exactly are we updating the weights of our target network? In particular, if we've already reached convergence, ...
0
votes
0answers
31 views

How is centralised training and decentralised execution in multi agent reinforcement learning implemented?

In the paper Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning, it is written We allow centralised training but require decentralised execution, from which follows that the policies ...
2
votes
1answer
79 views

What is the bias-variance trade-off in reinforcement learning?

I am watching DeepMind's video lecture series on reinforcement learning, and when I was watching the video of model-free RL, the instructor said the Monte Carlo methods have less bias than temporal-...
1
vote
1answer
38 views

What would happen if we sampled only one tuple from the experience replay?

The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right? What would happen if we calculate our ...
2
votes
1answer
69 views

Two DQNs in two different time scales

I have the following situation. An agent plays a game and wants to maximize the accumulated reward as usual, but it can choose its adversary. There are $n$ adversaries. In episode $e$, the agent must ...
1
vote
1answer
144 views

In Deep Q-learning, are the target update frequency and the batch training frequency related?

In a Deep Q-learning algorithm, we perform a batch training every train_freq and we update the parameters of the target network every ...
0
votes
0answers
44 views

Model Based rl and cross entropy method with nonlinear function approximators

Pseudo code for Cross entropy method according to youtube lecture 32:55 Initialize $\mu \in R^{d}, \sigma \in R^{d}$ iteration 1,2,... Collect n samples of $\theta_{i} \sim N(\mu,diag(\sigma))$ ...
0
votes
2answers
102 views

When we use a neural network to approximate the Q values, is the Q target a single value?

I have two questions When we use our network to approximate our Q values, is the Q target a single value? During backpropagation, when the weights are updated, does it automatically update the Q ...
2
votes
0answers
41 views

If the minimum Q value is decreasing and the maximum Q value increasing, is this a sign that dueling double DQN is diverging?

I'm training a dueling double DQN agent with prioritized replay buffer and notice that the min Q values are decreasing, while the max Q values are increasing. Is this a sign that it is diverging? ...
1
vote
1answer
91 views

How to implement RAM versions of Atari games

I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in ...