Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

If we have a working reward function, would adding another action have a significant effect on the agent performance if task remains the same?

If we have a working reward function, providing the desired behavior and optimal policy in a continuous action/state-space problem, would adding another action significantly affect the possible ...
user avatar
0 votes
0 answers
14 views

How to inform the autograd, my network structure includes RNN or LSTM? [closed]

Among all the Neural Network structures that are introduced, RNN has received noticeable attention because of the state art included in its gradient computation with backpropagation. On the other hand,...
user avatar
  • 1
2 votes
0 answers
52 views

How to reduce the dimensionality of the actions in RL

I have a single-agent RL model in which the dimension of the dimension of the action space is $70$. This action space is too big and the deep RL agent is not learning properly. The boundaries of the ...
user avatar
  • 49
0 votes
0 answers
19 views

When Hindsight Experience Replay is deployed, does the input need to be augmented as well?

In Hindsight Experience Replay (HER), we augment the state representation $s_t$ with some goal $g_T$, which corresponds to the state reached after $T$ steps, such that $s'_t = s_t || g_T$. Later, some ...
user avatar
  • 71
1 vote
1 answer
52 views

How should I write the reward function to teach the agent the rules of this card game?

I'm quite new to reinforcement learning. I've been training the model for the following problem but the mean reward is stuck. In a 5 by 5 board, each position can contain a card with a color (0-4) ...
user avatar
  • 111
1 vote
0 answers
39 views

What is commonly done for standardization/normalization of the targets in Deep Q-Learning?

I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
user avatar
  • 23
2 votes
1 answer
165 views

Determining to terminate at a reward or not

I am practicing the Bellman equation on Grid world examples and in this scenario, there are numbered grid squares where the agent can choose to terminate and collect the reward equal to the amount ...
user avatar
  • 123
0 votes
0 answers
21 views

How do you determine the optimal policy?

I am following some Grid world examples to understand reinforcement learning. I have a deterministic grid (part of which I have reconstructed below). I am trying to understand how the optimal policy ...
user avatar
  • 1
0 votes
0 answers
10 views

How are previous values updated when performing value iteration?

I have been trying to understand how you determine the value for each square in a grid world and I have seen/watched a few different examples to try and apply it to my own grid and I find myself ...
user avatar
  • 1
0 votes
1 answer
49 views

When calculating the max in DQN, do I have to calculate the Q for every possible action for a particular state?

I'm trying to implement the DQN paper using python/pytorch for my needs (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). I'm studying the main algorithm: I am a bit confused about the $\gamma* \max ...
user avatar
0 votes
0 answers
24 views

Determine Gridworld values

I am learning Reinforcement learning for games following Gridworld examples. Apologies in advance if this is a basic question, very new to reinforcement learning. I am slightly confused in scenarios ...
user avatar
  • 123
0 votes
0 answers
23 views

Can DDPG algorithm obtain time-dependent and time-independent actions simultaneously?

I am new to Reinforcement Learning. I have been working on a problem using Deep Deterministic Policy Gradient (DDPG). I would like to know if it is possible to apply this algorithm to an optimization ...
user avatar
0 votes
0 answers
35 views

What can we learn from AlphaZero in the development towards AGI?

According to DeepMind, AlphaZero's creative insights coupled with the encouraging results we see in other projects such as AlphaFold, give us confidence in our mission to create general purpose ...
user avatar
  • 111
1 vote
1 answer
45 views

How can AlphaZero be used in other industries besides gaming?

I'm an AI Engineering student from Belgium and I'm writing my bachelor thesis on the creation of a chess computer with deep reinforcement learning based on AlphaZero. My implementation can be found ...
user avatar
  • 111
3 votes
1 answer
60 views

How do I design the network for Deep Q-Network?

I am playing with deep q-learning and I am thinking about what the proper architecture of a network used for deep q-learning is. I have a very simple environment, basically a 18x18 matrix, where 3 ...
user avatar
  • 133
1 vote
1 answer
50 views

Doesn't the n-step Tree Backup algorithm negatively affect the DQN-Agent by creating inconsistent look-ahead targets?

In the text book of Sutton and Barto on page 152 they introduce the n-step Tree Backup algorithm, where the tree-backup n-step return is defined via $$ G_{t:t+n} = R_{t+1} + \gamma \sum_{a \neq A_{t+1}...
user avatar
  • 23
0 votes
0 answers
18 views

How to choose the best RL algorithm for a 10x5 action space?

I have RL problem statement where I am training a bot to observe a 350x10x10 matrix and expect a 10x5 action vector The observation space is sort of a time series ...
user avatar
3 votes
2 answers
337 views

What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?

In Deep Reinforcement Learning (DRL) I am having difficulties in understanding the difference between a Loss function, a reward/penalty and the integration of both in DRL. Loss function: Given an ...
user avatar
  • 131
1 vote
1 answer
32 views

Can directly using expert policy in epsilon-greedy speed-up Q-learning?

In deep Q-learning we typically use epsilon-greedy policy during training. We choose a random action for a certain probability $\epsilon$, and choose the action that maximize the current Q-value ...
user avatar
  • 173
1 vote
1 answer
30 views

Is using Monte-Carlo estimate of returns in Deep Q Learning possible?

In all the tutorials of deep Q-learning (using neural networks) I have read so far, the state-action value function $Q(s,a)$ is learned by temporal difference learning. However, in policy gradient ...
user avatar
  • 173
1 vote
0 answers
37 views

How to calculate uncertainty in Deep Ensembles for Reinforcement Learning?

Lets take the following example: I must predict the return (Q-values) of x state-action pairs using an ensemble of m models. Using NumPy I could have the following for x = 5 and m = 3: ...
user avatar
  • 71
1 vote
1 answer
79 views

How to take gradient of log policy when actions are negative?

I am currently trying to train BipedalWalker of OpenAI gym by using policy gradient approach. My action space contains 4 continuous actions, all ranging [-1.0, 1.0]. In this case, how can we calculate ...
user avatar
0 votes
1 answer
44 views

How to define actions on a list of values?

For a DQN algorithm, where my state is a list of values, say: [5, 3, 4, 7, 8, 2, 6] How can I define an action space that allows me to move a value in the list from one position to another? For ...
user avatar
  • 3
1 vote
0 answers
45 views

Alternatives to using RL to explore an environment

I am looking at some ideas on exploring an environment using Curiosity Driven Exploration and am being a little skeptical about it. The objective here is to just explore without the need to obtain ...
user avatar
2 votes
0 answers
31 views

How to deal with variable action ranges in RL for continuous action spaces

I am reading this paper on battery management using RL. The action consist in the charging/discharging power of the battery at timestep $t$. For instance, in the case of the charging power, the ...
user avatar
  • 49
0 votes
1 answer
51 views

Why does my actor-critic network always give either -1 or 1 at the output layer?

I have an actor-critic network. The state space contains continuous variables with different ranges like (0,1.57) and (-0.70, 0.70). And it also contain absolute 6D pose in the form (x,y,z,roll,pitch,...
user avatar
1 vote
0 answers
39 views

Are tabular methods appropriate for this task?

I'm a RL beginner and I have a project in mind: I'm an engineer who works doing CAD projects and there is a time consuming task that I want to automatize. This task it's doing by a person because it ...
user avatar
  • 111
0 votes
1 answer
101 views

Action selection in Batch-Constrained Deep Q-learning (BCQ)

For simplicity, let's consider the discrete version of BCQ where the paper and the code are available. In the line 5 of Algorithm 1 we have the following: $$ a' = \text{argmax}_{a'|G_{\omega}(a', s')/\...
user avatar
  • 71
0 votes
0 answers
31 views

Training a RL agent using different data at each episode

I am training a RL agent whose state is composed of two numbers, ranging between 4 ~ 16 and 0 ~ 360. The action is continuous and between 0~90. In real life, the states can be any I am training a TD3 ...
user avatar
  • 49
0 votes
1 answer
93 views

How should I initialize the weights of the neural network so that the initial policy is uniform?

I would like to train a neural network (NN) so that it learns the policy and value function for my agent. Since I am using reinforcement learning and do not want to prefer certain actions in certain ...
user avatar
  • 159
0 votes
1 answer
33 views

How Come My (D)DQN Fails To Learn?

I am currently trying to teach a (D)DQN algorithm to play a 10x10 GridWorld game, so I can compare the two as I increase the number of moves the agent can take. The rewards are as follows: A step = -1 ...
user avatar
2 votes
2 answers
105 views

Does AlphaGo play random moves in a real competition?

Alphago and AlphaGo zero use random play to generate data and use the data to train DNN. "Random play" means that there is a positive probability for AlphaGo to play some suboptimal moves ...
user avatar
  • 141
1 vote
0 answers
15 views

Output representation for a neural network to learn grid-based game with multiple units

I have a round based game played on a grid map with multiple units that I would like to control in some fashion using neural network (NN). All of the units are moved at once. Each unit can move in any ...
user avatar
  • 159
0 votes
0 answers
18 views

How to reduce the variance of stochastic policy gradient for continuous actions in a partially observable environment?

I am trying to implement a stochastic policy gradient for continuous actions in a partially observable CartPole environment. Specifically, only the current cart position and pole angle are visible, ...
user avatar
  • 173
0 votes
0 answers
50 views

How to pass the rewards in zero-sum multiplayer context when using REINFORCE?

Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm. I have doubts about passing reward values in ...
user avatar
  • 2,977
1 vote
1 answer
55 views

Is it the high probability action that is always selected by the agent in REINFORCE algorithm?

Consider the following algorithm from the textbook titled Reinforcement Learning: An Introduction (second edition) by Richard S. Sutton and Andrew G. Bart While playing the game for the generation of ...
user avatar
  • 2,977
0 votes
1 answer
53 views

Why is old/off-policy data harmful to on-policy/online RL? [closed]

I ask because if RL is indeed an MDP, then there should be absolutely no problem with training an agent on any available episode roll-out data, right? Because an MDP implies for any state S, the ...
user avatar
  • 266
1 vote
0 answers
86 views

What method is better to use for a two-player reinforcement learning environment?

I want to create an RL agent for a mancala-type two-player game as my first actual project in the field. I've already completed the game itself and coded a minimax algorithm. The question is: how ...
user avatar
1 vote
0 answers
37 views

Proper way to count environment steps / frames in distributed RL architecture for algorithms like CLEAR or LASER => modified impala with replay

In classical - on-policy - vtrace/Impala algorithm env_steps are incremented every training iteration like this : ...
user avatar
1 vote
0 answers
22 views

Why Acme is using own uniform initializer?

Why is Acme using own initializer for both tanh and ELU, when commonly used for tanh is Xavier and for ELU is He initializer? What mathematics is behind them? Here is the code. ...
user avatar
0 votes
0 answers
39 views

Should you clip Q values if they start to grow indefinitely?

I am training the SAC algorithm for an environment where the rewards are small as shown below and the episode length is 84. I have a problem with the Q values that grow with each step. The following ...
user avatar
  • 167
0 votes
0 answers
56 views

Deep Reinforcement Learning - no optimizing of cumulated rewards

I have a Deep Reinforcement Learning code for trading with a trainer which has 20 possible actions (profit taking from 0% "holding" to -100% "close position with -5% increments). States ...
user avatar
0 votes
0 answers
21 views

Generate gaming sequences of inputs like how GANs do for art

I need some insight on a subject given the fact that I'm not a researcher, but I am a software engineer. I want to build a model that would recommend (or generate) different paths (play sequences) in ...
user avatar
0 votes
0 answers
29 views

IQN being outperformed by DDQN. Posible reasons?

I have been getting into RL, and I have a DDQN model that learns how to play the super mario 1-1 world. Then, I tried using the code from the IQN paper to play this game (modified the DQN" part ...
user avatar
0 votes
0 answers
103 views

tfp.Distributions.Categorical.sample() is picking the same action everytime after certain iterations

I have written a code for an RL agent such that at each state the model calculates the probabilities of all possible actions and samples one action randomly to proceed further. To acheive this, I have ...
user avatar
0 votes
0 answers
56 views

FrozenLake-v0 not training using REINFORCE

I am implementing a simple REINFORCE (policy gradient) algorithm for openAI's FrozenLake-v0 environment. However, it does not seem to learn anything at all. I have used the same neural architecture ...
user avatar
  • 101
1 vote
0 answers
59 views

Can Q-learning be used for my scenario, and how might I do so?

I have already asked 2-3 general questions w.r.t Q learning and now I am asking a scenario specific one. I will try to be concise and understandable. I really really need help. Scenario: I have a ...
user avatar
-2 votes
1 answer
61 views

Is it possible to train an RL agent using images?

I have an image which consists of a start and an end point, the journey has some obstacles which have to be avoided. Is it possible to train an RL agent using such images to find the best path ...
user avatar
-1 votes
1 answer
54 views

What does the line of code "self.buffer[-1] = observation" do in this BufferWrapper class for DQN?

So the code is related to using a buffer ...
user avatar
0 votes
0 answers
33 views

How to deal with Q-learning having low variance in predicted Q-values?

I have a neural network that takes the state (which contains a lot of data), and the possible action (which is very little data), and predicts the Q-value of the action. I am double Q-learning. I've ...
user avatar
  • 266

1
2 3 4 5
8