Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

What reinforcement learning architecture is recommended for multiple outputs in continuous resource management?

Brief background: I would like to develop an agent to provide resources to multiple machines simultaneously. The overall resources are limited. The agent should distribute the resources in such a way ...
user avatar
0 votes
0 answers
8 views

Correct reward for trading applications. RL agent learns according to immediate reward instead of cumulative reward

I have coded a RL environment for trading. The action space is discrete with 3 components [0,1,2]; where 0 corresponds to selling an amount of shares; 1 corresponds to holding; and 2 corresponds to ...
user avatar
  • 49
0 votes
1 answer
69 views

How do I create an AI controller for Pacman? [closed]

How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game? It ...
user avatar
  • 11
1 vote
2 answers
34 views

How many training steps does it usually take to train an RL model?

This is my model average rewards as follow image. How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model? And I'm using PPO to train.
user avatar
  • 123
0 votes
1 answer
48 views

How to compare memory requirements for tabular Q-learning vs deep neural network?

I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...
user avatar
0 votes
0 answers
26 views

How to get into robotic simulation for RL purposes? [closed]

For my master's thesis, I've joined a robotics team that tries to build a flying robot based on the mechanism of flies (i.e - two wings that perform reciprocating motion to generate lift). My part in ...
user avatar
0 votes
0 answers
20 views

Why is it difficult to train large RL networks?

First of all I know that: 'it makes training less stable' & 'RL is already inherently unstable'. I'm asking why those things are true? Intuitively it seems very strange & to be perhaps a ...
user avatar
  • 276
0 votes
0 answers
25 views

Gym like reinforcement learning frameworks for combinatorial optimization on Graphs [closed]

I was wondering if anyone knows of a gym like framework for combinaotrial optimization with reinforcement learning, which deals with max-cut, travelling sales person problem and other interesting ...
user avatar
1 vote
1 answer
28 views

How to preserve Markov Property in Deep Reinforcement Learning when using "mixup" or "mixreg"?

I've read through these two papers: (original about "mixup") https://arxiv.org/pdf/1710.09412.pdf (variant for RL, "mixreg") https://arxiv.org/pdf/2010.10814.pdf They are about a ...
user avatar
1 vote
1 answer
48 views

How to tune hypeparametes in A2C-ppo?

Im currently working with A2C. The model was able to learn open ai pong, i ran this as a sanity check that i havent made any bugs. Now im trying to make the model play breakout, but still after 10m ...
user avatar
3 votes
1 answer
51 views

Why is a large replay buffer inefficient?

Open AI spin up says ... the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. If you only use the very-most recent data, ...
user avatar
  • 131
1 vote
1 answer
22 views

How to implement PPO without using a Critic

I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my ...
user avatar
  • 13
0 votes
0 answers
43 views

RL-based trading bot: how to deal with overfitting

I've been playing around building a reinforcement learned-based trading bot using the stable-baselines3 library. I've come up with an environment that seems to be able to learn how to make profitable ...
user avatar
  • 101
0 votes
0 answers
9 views

Transformer-XL query length differs during inference and optimization?

I'm working through KakaoBrain's Transformer Reinforcement Learning implementation. https://github.com/kakaobrain/brain_agent I observed that the query length during sampling is set to ...
user avatar
0 votes
1 answer
33 views

How to deal with changing rewards in Q-learning? DQN?

I read the working of Q-learning through a grid-based taxi routing wherein a taxi has to pick and drop off a passenger from source to destination. Likewise, I have a routing problem and hence, I tried ...
user avatar
0 votes
0 answers
26 views

Why are agents trained in episodes, even in non-episodic tasks?

Let's consider some non-episodic problem. Maybe a game which can go on forever. My question is: Why are agents still trained in episodes? My understanding is that the agent's neural network is updated ...
user avatar
4 votes
1 answer
927 views

Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

I want to understand roughly what is happening in the neural network of an RL agent when it is exploring vs. exploiting. For example, are the network weights not being updated when the agent is ...
user avatar
0 votes
1 answer
52 views

Why aren't neural networks contractions?

I'm not sure I understand why neural networks aren't considered contractions, as Geoffrey J. Gordon says in his paper: Stable Function Approximation in Dynamic Programming: "Our theorems in the ...
user avatar
1 vote
0 answers
19 views

Is it possible to combine two policy-based RL agents?

I am developing an RL agent for a game environment. I have found out that there are two strategies to do well in the game. So I have trained two RL agents using neural networks with distinct reward ...
user avatar
1 vote
1 answer
35 views

In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?

In the context of my problem, the "true" reward is not additive. Realistically, the more reward the agent has already accumulated, the easier it becomes to accumulate even more. That's to ...
user avatar
1 vote
0 answers
39 views

Policy gradient (or more general, RL algorithms) for the problems where actions does not determine next state (next state is independent to action)

I am pretty new in RL. Could anyone suggest results/paper about whether or not policy gradient (or more general RL algorithms) can be applied to the problems where actions does not determine next ...
user avatar
  • 11
0 votes
1 answer
37 views

How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?

For the back weight update step, I need to calculate $\nabla\hat{q}(S,A,w)$. My neural network takes in the state vector $S$ and gives out the action values for state $S$ and each action in the action ...
user avatar
2 votes
1 answer
130 views

Why do we use "true labels" that are based on the output of our network in Deep Q-Learning?

In the original DQN paper, the $\ell_2$ loss is taken over the distance between our network output, $\hat{q}(s_j,a_j,w)$ and the labels $y_j=r_j+\gamma \cdot \max\limits_{a'} \hat{q}(s_{j+1},a',w^-)$, ...
user avatar
2 votes
3 answers
107 views

Do I need to normalize all state-space variables? If so, how?

I am playing around with a DRL agent in a stock-trading environment. I have normalized all the external input data (the features that my agent will use). However, what about characteristics that don't ...
user avatar
0 votes
0 answers
14 views

DDQN for Connect 4: Sudden explosion of Loss

I am trying to solve Connect 4 with DDQN through the self-play regime that was used for AlphaZero. That means, I let a student version play against a teacher version of itself and replace the teacher ...
user avatar
  • 31
2 votes
1 answer
93 views

Why does the average-reward estimator for continuing tasks use the TD error?

In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q_\pi(s,a)$ and value function $v_\pi(s)$ ...
user avatar
  • 66
2 votes
0 answers
26 views

Is Reinforcement Learning with only feedback on a single action possible?

Consider the following case: A reinforcement based web-crawler where: State = current page + 1 out-link (reduced to features of some sort) Action = Whether to visit that out-link or not (n_actions = ...
user avatar
  • 175
0 votes
1 answer
21 views

Replay buffer action range in DDPG

I have an environment where the agent action is in range [0, 1.57]. My actor network in DDPG has a tanh activation, and so the network values are in the range [-1,1]. Hence I change the scaling from [-...
user avatar
1 vote
0 answers
20 views

If we have a working reward function, would adding another action have a significant effect on the agent performance if task remains the same?

If we have a working reward function, providing the desired behavior and optimal policy in a continuous action/state-space problem, would adding another action significantly affect the possible ...
user avatar
  • 11
2 votes
0 answers
55 views

How to reduce the dimensionality of the actions in RL

I have a single-agent RL model in which the dimension of the dimension of the action space is $70$. This action space is too big and the deep RL agent is not learning properly. The boundaries of the ...
user avatar
  • 49
0 votes
0 answers
23 views

When Hindsight Experience Replay is deployed, does the input need to be augmented as well?

In Hindsight Experience Replay (HER), we augment the state representation $s_t$ with some goal $g_T$, which corresponds to the state reached after $T$ steps, such that $s'_t = s_t || g_T$. Later, some ...
user avatar
  • 81
1 vote
1 answer
61 views

How should I write the reward function to teach the agent the rules of this card game?

I'm quite new to reinforcement learning. I've been training the model for the following problem but the mean reward is stuck. In a 5 by 5 board, each position can contain a card with a color (0-4) ...
user avatar
  • 111
1 vote
0 answers
51 views

What is commonly done for standardization/normalization of the targets in Deep Q-Learning?

I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
user avatar
  • 23
2 votes
1 answer
170 views

Determining to terminate at a reward or not

I am practicing the Bellman equation on Grid world examples and in this scenario, there are numbered grid squares where the agent can choose to terminate and collect the reward equal to the amount ...
user avatar
  • 123
0 votes
0 answers
30 views

How do you determine the optimal policy?

I am following some Grid world examples to understand reinforcement learning. I have a deterministic grid (part of which I have reconstructed below). I am trying to understand how the optimal policy ...
user avatar
  • 1
0 votes
0 answers
10 views

How are previous values updated when performing value iteration?

I have been trying to understand how you determine the value for each square in a grid world and I have seen/watched a few different examples to try and apply it to my own grid and I find myself ...
user avatar
  • 1
0 votes
1 answer
139 views

When calculating the max in DQN, do I have to calculate the Q for every possible action for a particular state?

I'm trying to implement the DQN paper using python/pytorch for my needs (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). I'm studying the main algorithm: I am a bit confused about the $\gamma* \max ...
user avatar
0 votes
0 answers
26 views

Determine Gridworld values

I am learning Reinforcement learning for games following Gridworld examples. Apologies in advance if this is a basic question, very new to reinforcement learning. I am slightly confused in scenarios ...
user avatar
  • 123
0 votes
0 answers
29 views

Can DDPG algorithm obtain time-dependent and time-independent actions simultaneously?

I am new to Reinforcement Learning. I have been working on a problem using Deep Deterministic Policy Gradient (DDPG). I would like to know if it is possible to apply this algorithm to an optimization ...
user avatar
1 vote
1 answer
58 views

What can we learn from AlphaZero in the development towards AGI?

According to DeepMind, AlphaZero's creative insights coupled with the encouraging results we see in other projects such as AlphaFold, give us confidence in our mission to create general purpose ...
user avatar
  • 123
1 vote
1 answer
52 views

How can AlphaZero be used in other industries besides gaming?

I'm an AI Engineering student from Belgium and I'm writing my bachelor thesis on the creation of a chess computer with deep reinforcement learning based on AlphaZero. My implementation can be found ...
user avatar
  • 123
3 votes
1 answer
69 views

How do I design the network for Deep Q-Network?

I am playing with deep q-learning and I am thinking about what the proper architecture of a network used for deep q-learning is. I have a very simple environment, basically a 18x18 matrix, where 3 ...
user avatar
  • 133
1 vote
1 answer
67 views

Doesn't the n-step Tree Backup algorithm negatively affect the DQN-Agent by creating inconsistent look-ahead targets?

In the text book of Sutton and Barto on page 152 they introduce the n-step Tree Backup algorithm, where the tree-backup n-step return is defined via $$ G_{t:t+n} = R_{t+1} + \gamma \sum_{a \neq A_{t+1}...
user avatar
  • 23
0 votes
0 answers
20 views

How to choose the best RL algorithm for a 10x5 action space?

I have RL problem statement where I am training a bot to observe a 350x10x10 matrix and expect a 10x5 action vector The observation space is sort of a time series ...
user avatar
3 votes
2 answers
529 views

What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?

In Deep Reinforcement Learning (DRL) I am having difficulties in understanding the difference between a Loss function, a reward/penalty and the integration of both in DRL. Loss function: Given an ...
user avatar
  • 131
1 vote
1 answer
41 views

Can directly using expert policy in epsilon-greedy speed-up Q-learning?

In deep Q-learning we typically use epsilon-greedy policy during training. We choose a random action for a certain probability $\epsilon$, and choose the action that maximize the current Q-value ...
user avatar
  • 173
1 vote
1 answer
31 views

Is using Monte-Carlo estimate of returns in Deep Q Learning possible?

In all the tutorials of deep Q-learning (using neural networks) I have read so far, the state-action value function $Q(s,a)$ is learned by temporal difference learning. However, in policy gradient ...
user avatar
  • 173
1 vote
0 answers
41 views

How to calculate uncertainty in Deep Ensembles for Reinforcement Learning?

Lets take the following example: I must predict the return (Q-values) of x state-action pairs using an ensemble of m models. Using NumPy I could have the following for x = 5 and m = 3: ...
user avatar
  • 81
1 vote
1 answer
82 views

How to take gradient of log policy when actions are negative?

I am currently trying to train BipedalWalker of OpenAI gym by using policy gradient approach. My action space contains 4 continuous actions, all ranging [-1.0, 1.0]. In this case, how can we calculate ...
user avatar
0 votes
1 answer
45 views

How to define actions on a list of values?

For a DQN algorithm, where my state is a list of values, say: [5, 3, 4, 7, 8, 2, 6] How can I define an action space that allows me to move a value in the list from one position to another? For ...
user avatar
  • 3

1
2 3 4 5
8