# Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

402 questions
Filter by
Sorted by
Tagged with
5 views

### How to quickly train your agent to get a sense of the problem?

I have a custom implementation of DQN. My robot/agent is running in Gazebo with ROS. Though I am trying a very simple task of pushing a cube, the DQN agent is taking too much time. DQN of 300 episode ...
15 views

### Why does mean episode reward during training differ dramatically from "manual" runs of the trained model on same data?

I am training an RL agent, using PPO, on a time-series environment that comes from a tabular dataset. The possible scores during an episode goes from -1 to positive infinity (though realistically, I ...
1 vote
24 views

### Is it possible for AlphaGo Zero to use recurrent networks to achieve similar performance?

AlphaGo Zero stacks 7 board history along with the current board together to form the input to the network. However, is it possible to use an RNN to replace the input of history and achieve similar ...
• 183
1 vote
19 views

### How to teach DRL agent to avoid erratic, high frequency behavior (when costs for taking actions are already included)?

I have a situation where an agent can take actions to enter a state where rewards can be obtained (or costs can be incurred), but the actions themselves have a small cost. In my reward function, I ...
18 views

### What reinforcement learning architecture is recommended for multiple outputs in continuous resource management?

I would like to develop an agent to provide resources to multiple machines simultaneously. The overall resources are limited. The agent should distribute the resources in such a way that the machines ...
8 views

### Correct reward for trading applications. RL agent learns according to immediate reward instead of cumulative reward

I have coded a RL environment for trading. The action space is discrete with 3 components [0,1,2]; where 0 corresponds to selling an amount of shares; 1 corresponds to holding; and 2 corresponds to ...
• 49
84 views

### How do I create an AI controller for Pacman?

How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game? It ...
• 11
1 vote
35 views

### How many training steps does it usually take to train an RL model?

This is my model average rewards as follow image. How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model? And I'm using PPO to train.
• 123
50 views

### How to compare memory requirements for tabular Q-learning vs deep neural network?

I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...
26 views

### How to get into robotic simulation for RL purposes? [closed]

For my master's thesis, I've joined a robotics team that tries to build a flying robot based on the mechanism of flies (i.e - two wings that perform reciprocating motion to generate lift). My part in ...
21 views

### Why is it difficult to train large RL networks?

First of all I know that: 'it makes training less stable' & 'RL is already inherently unstable'. I'm asking why those things are true? Intuitively it seems very strange & to be perhaps a ...
• 276
1 vote
28 views

### How to preserve Markov Property in Deep Reinforcement Learning when using "mixup" or "mixreg"?

I've read through these two papers: (original about "mixup") https://arxiv.org/pdf/1710.09412.pdf (variant for RL, "mixreg") https://arxiv.org/pdf/2010.10814.pdf They are about a ...
1 vote
48 views

### How to tune hypeparametes in A2C-ppo?

Im currently working with A2C. The model was able to learn open ai pong, i ran this as a sanity check that i havent made any bugs. Now im trying to make the model play breakout, but still after 10m ...
53 views

### Why is a large replay buffer inefficient?

Open AI spin up says ... the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. If you only use the very-most recent data, ...
• 131
1 vote
25 views

### How to implement PPO without using a Critic

I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my ...
• 13
43 views

### RL-based trading bot: how to deal with overfitting

I've been playing around building a reinforcement learned-based trading bot using the stable-baselines3 library. I've come up with an environment that seems to be able to learn how to make profitable ...
• 101
9 views

### Transformer-XL query length differs during inference and optimization?

I'm working through KakaoBrain's Transformer Reinforcement Learning implementation. https://github.com/kakaobrain/brain_agent I observed that the query length during sampling is set to ...
• 101
33 views

### How to deal with changing rewards in Q-learning? DQN?

I read the working of Q-learning through a grid-based taxi routing wherein a taxi has to pick and drop off a passenger from source to destination. Likewise, I have a routing problem and hence, I tried ...
27 views

### Why are agents trained in episodes, even in non-episodic tasks?

Let's consider some non-episodic problem. Maybe a game which can go on forever. My question is: Why are agents still trained in episodes? My understanding is that the agent's neural network is updated ...
928 views

### Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

I want to understand roughly what is happening in the neural network of an RL agent when it is exploring vs. exploiting. For example, are the network weights not being updated when the agent is ...
52 views

### Why aren't neural networks contractions?

I'm not sure I understand why neural networks aren't considered contractions, as Geoffrey J. Gordon says in his paper: Stable Function Approximation in Dynamic Programming: "Our theorems in the ...
1 vote
19 views

### Is it possible to combine two policy-based RL agents?

I am developing an RL agent for a game environment. I have found out that there are two strategies to do well in the game. So I have trained two RL agents using neural networks with distinct reward ...
• 111
1 vote
35 views

### In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?

In the context of my problem, the "true" reward is not additive. Realistically, the more reward the agent has already accumulated, the easier it becomes to accumulate even more. That's to ...
1 vote
40 views

### Policy gradient (or more general, RL algorithms) for the problems where actions does not determine next state (next state is independent to action)

I am pretty new in RL. Could anyone suggest results/paper about whether or not policy gradient (or more general RL algorithms) can be applied to the problems where actions does not determine next ...
• 11
37 views

### How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?

For the back weight update step, I need to calculate $\nabla\hat{q}(S,A,w)$. My neural network takes in the state vector $S$ and gives out the action values for state $S$ and each action in the action ...
130 views

### Why do we use "true labels" that are based on the output of our network in Deep Q-Learning?

In the original DQN paper, the $\ell_2$ loss is taken over the distance between our network output, $\hat{q}(s_j,a_j,w)$ and the labels $y_j=r_j+\gamma \cdot \max\limits_{a'} \hat{q}(s_{j+1},a',w^-)$, ...
118 views

### Do I need to normalize all state-space variables? If so, how?

I am playing around with a DRL agent in a stock-trading environment. I have normalized all the external input data (the features that my agent will use). However, what about characteristics that don't ...
14 views

### DDQN for Connect 4: Sudden explosion of Loss

I am trying to solve Connect 4 with DDQN through the self-play regime that was used for AlphaZero. That means, I let a student version play against a teacher version of itself and replace the teacher ...
• 31
94 views

### Why does the average-reward estimator for continuing tasks use the TD error?

In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q_\pi(s,a)$ and value function $v_\pi(s)$ ...
• 66
26 views

### Is Reinforcement Learning with only feedback on a single action possible?

Consider the following case: A reinforcement based web-crawler where: State = current page + 1 out-link (reduced to features of some sort) Action = Whether to visit that out-link or not (n_actions = ...
• 175
23 views

### Replay buffer action range in DDPG

I have an environment where the agent action is in range [0, 1.57]. My actor network in DDPG has a tanh activation, and so the network values are in the range [-1,1]. Hence I change the scaling from [-...
1 vote
20 views

### If we have a working reward function, would adding another action have a significant effect on the agent performance if task remains the same?

If we have a working reward function, providing the desired behavior and optimal policy in a continuous action/state-space problem, would adding another action significantly affect the possible ...
• 11
55 views

### How to reduce the dimensionality of the actions in RL

I have a single-agent RL model in which the dimension of the dimension of the action space is $70$. This action space is too big and the deep RL agent is not learning properly. The boundaries of the ...
• 49
24 views

### When Hindsight Experience Replay is deployed, does the input need to be augmented as well?

In Hindsight Experience Replay (HER), we augment the state representation $s_t$ with some goal $g_T$, which corresponds to the state reached after $T$ steps, such that $s'_t = s_t || g_T$. Later, some ...
• 81
1 vote
62 views

### How should I write the reward function to teach the agent the rules of this card game?

I'm quite new to reinforcement learning. I've been training the model for the following problem but the mean reward is stuck. In a 5 by 5 board, each position can contain a card with a color (0-4) ...
• 111
1 vote
51 views

### What is commonly done for standardization/normalization of the targets in Deep Q-Learning?

I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
• 23
170 views

### Determining to terminate at a reward or not

I am practicing the Bellman equation on Grid world examples and in this scenario, there are numbered grid squares where the agent can choose to terminate and collect the reward equal to the amount ...
• 123
31 views

### How do you determine the optimal policy?

I am following some Grid world examples to understand reinforcement learning. I have a deterministic grid (part of which I have reconstructed below). I am trying to understand how the optimal policy ...