Questions tagged [deep-rl]
For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.
499
questions
1
vote
2
answers
26
views
How to handle the dead agent in multi-agent environment?
I recently try to implement deep reinforcement learning on a defender-vs-attacker problem, where agents can be destroyed by enemies. I am coding both the environment and the RL algorithm. The agent ...
- 11
0
votes
0
answers
21
views
Deep Reinforcement Learning that takes action from two different sets
I am working on a problem where I want to schedule multiple activities (a1, a2, a3, ... aN) requiring different resource types ...
- 1
0
votes
1
answer
19
views
Sequential models and distribution shift in RL
We know the problem of "distribution shift" in deep Reinforcement Learning, where the change in policy during training affects the behavior of the agent and therefore the distribution of the ...
1
vote
0
answers
21
views
Why policy gradient theorem has two different forms?
I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton &...
- 11
0
votes
0
answers
19
views
Policy Gradient in Partial Observability
Let $\pi_{\theta}$ be a policy. Then, I was able to follow through the proof of: $\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|s_i)R(\tau)]$, where ...
- 1
0
votes
1
answer
35
views
Understanding KL Stopping and KL Cutoff for the PPO algorithm
I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the ...
- 588
1
vote
0
answers
13
views
How does recurrent neural network implement model based RL system purely in its activation dynamics (in blackbox meta-rl setting)?
I have read these papers "learning to reinforcement learn" and "PFC as meta RL system". The authors claim that when RNN is trained on multiple tasks from a task distribution using ...
- 19
1
vote
1
answer
92
views
What kind of observation state would you give for that environment?
I'm making a new environment where I have two sphere (one above the other) in a 2D plan.
I would like some advice on what observation state I should give to my RL.
Today I have given the following:
...
- 135
1
vote
1
answer
35
views
why learn an observation model when training latent space model in model based rl
I'm currently studying reinforcement learning through CS 285 provided by UC Berkeley.
At 1:52 of the part 5 of the lecture 11, I got confused on why one would want to learn an observation model $p(o_t ...
- 13
0
votes
0
answers
41
views
Using deep reinforcement learning for malware detection; trained agent mostly performs the same action
I'm trying to implement this article:
Ransomware early detection using deep reinforcement learning on portable executable header
The article uses an unpublished dataset of benign and ransomware ...
0
votes
0
answers
13
views
Proof of Difference in Return Between Two Policies
I am attempting to understand why Lemma 6.1 holds in this paper on reinforcement learning. I have two questions. First, when defining the value function V(s), why is there a leading (1-γ) term? In the ...
0
votes
1
answer
74
views
Is my PPO agent learning? or is it just exploring?
I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src.
My doubts are mainly due ...
- 1
0
votes
0
answers
54
views
Question about feature matrix and notation in the paper Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
Let me know if this is not the place for this question. I'll take it down happily if that's the case. Also, I emailed the authors, but it seems like I won't be getting a response, so that's why I am ...
- 242
2
votes
2
answers
530
views
How do you deal with movement inertia in an environment after a step?
I was wondering how can we deal with movement inertia in an environment that is constantly changing?
Imagine that you make a step on an environment that moves a ball. When you make the step, you make ...
- 135
0
votes
0
answers
19
views
Changing Gym Environment Mid Training
I am using a custom gym environment for a research project. As my agent solves the environment, I want the task to get progressively harder. Right now, I am doing that like so:
...
0
votes
0
answers
12
views
Help defining environment with complex action space
I'm working on a personal MARL project with a high-dimensional and continuous action space. The environment is designed to give positive rewards to actions between some moving limits of the action ...
-1
votes
1
answer
39
views
RL framework to optimize my custom multi-agent simulator [closed]
I have built a custom discrete event simulator with multiple agents and want to optimize the system using RL frameworks that support multi-agent configurations. I will use custom policies. Which ...
0
votes
1
answer
23
views
Can I add additional arguments to my custom Gym Environment? [closed]
I have a custom gym (not gymnasium) environment that I am using for research. I am currently using gym version 0.19.0 installed using conda-forge. Glossing over a lot of details, the agent is learning ...
0
votes
1
answer
22
views
How does DeepQ learn for different environments?
I'm studying Deep Reinforcement Learning using the book 'DRL in Action' by Zai and Brown. In chapter 3, they present the classic GridWorld example, which can be randomly initialized. This means that ...
- 221
0
votes
1
answer
106
views
How do I add Entropy to a PPO algorithm?
I learned about adding entropy to RL algorithms through the notes provided in SpinningUp. They explained how entropy is added to the SAC algorithm. Here is my understanding - In entropy regularized RL,...
- 588
1
vote
1
answer
74
views
What do simulations mean in the context of the AlphaGoZero paper?
I am familiar with the Monte Carlo Tree Search, which can be broken down into 4 parts - Selection, Expansion, Simulation and Backpropogation. In this case, the simulation step denotes performing a ...
- 588
1
vote
1
answer
26
views
How to handle large dimensionality differences between state and action inputs in a reinforcement learning predictor?
I'm currently writing code for a reward predictor function r=f(s,a) in reinforcement learning, where 's' is the state with 256 dimensions (the embedding dimension after visual input is processed by an ...
- 41
0
votes
0
answers
10
views
When should grayscale processing be applied to image inputs in visual reinforcement learning environments?
I am currently working with visual environments in Reinforcement Learning (RL) and have noticed differing practices regarding preprocessing of image inputs. Specifically, in the Atari environment, a ...
- 41
1
vote
1
answer
47
views
Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?
I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in ...
- 41
0
votes
0
answers
31
views
How to tell an agent that some actions in the action space are currently not available in gym and the design of action space
I want to make a task allocation decision by reinforcement learning. Suppose there are N tasks to be allocated and M severs to serve these task.
However, there is a constraint that one task should be ...
- 1
1
vote
0
answers
23
views
Is there any Deep RL method that is based on value function approximation of Post-decision States
I am trying to construct an RL algorithm for managing a fleet of vehicles to maximize profit. As far as I know, the Sequential Decision Process can be decomposed into the following pic:
My current ...
- 111
0
votes
0
answers
32
views
Deep Q problem. Some q-Values are always greater
Description
My problem is to make actor play on stock market. I am trying to teach him playing on some portion of data. I interpolated data to uniformed interals and normalized or standarized all ...
0
votes
0
answers
60
views
What would be an appropriate way of making an AI for Cell Machine?
The game Cell Machine is absolutely by itself not suitable for AI usage, but a certain niche within a mod of the game (Mystic Mod) known as "vaults" does seem like a suitable use case for AI....
user74135
1
vote
1
answer
87
views
How would one normalize observations in off-policy online reinforcement learning?
In off-policy algorithms such as DQN, you need to feed your input to a network twice. 1. When inputting into a network for predicting the Q values. 2. When feeding the input from the buffer to the ...
- 588
0
votes
1
answer
27
views
Fit Q Evaluation in offline reinforcement learning
I am working on a PyTorch implementation of Implicit Q-Learning (IQL) (paper), given a dataset $\mathcal D = \left\{ (\mathbf s_i, \mathbf a_i, \mathbf s_i', r_i ) \right\}$ of transitions. I think I ...
- 101
1
vote
1
answer
128
views
How does one normalize observations in online reinforcement learning
I was wondering how would one normalize observations to a policy without knowing the upper and lower limits of the environment values. A trivial technique would be normalize each observation by its ...
- 588
2
votes
1
answer
187
views
Can I train an agent with DQN, avoiding obstacles and still finding the optimal path to finish the task?
The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible without ...
- 53
2
votes
1
answer
138
views
When using (s,a,r,s') to train networks could the Q network be adjusting to a suboptimal r?
My question here is that whenever you take an experience from the experience buffer (s,a,r,s') and you input r into r+yMAXQ(s',a') to get the loss. What if the r from that experience (s,a,r,s') is not ...
- 53
2
votes
0
answers
20
views
In DQN how does the Q network not converge to the incorrect target? [duplicate]
Whenever you are doing reinforcement learning you periodically update the target network based on the weights of the Q network. While I do understand this helps create a stable target I do not ...
- 53
0
votes
0
answers
29
views
MADDPG: reward drops fast after updating ac network
I'm using MADDPG to do computation offloading. But the reward drops fast once I start sampling from the replay buffer.
...
- 1
0
votes
0
answers
29
views
2D layout problem with RL - questions about state representation
I want to solve the following sort of problem with RL, specifically Q-learning (using the tensorflow-agents library):
The input is a polygon $\mathcal{P}$ and a list of rectangle sizes $\mathcal{L}=\{...
- 129
0
votes
0
answers
27
views
DDQN doesn't learn my modified version of Taxi-v3 from OpenAI
I tried to enhance the Taxi-v3 environment from OpenAI (true credit goes of course to https://arxiv.org/abs/cs/9905014) to be capable of handling multiple taxis/passengers. Don't worry, I'm not ...
- 101
3
votes
1
answer
235
views
Take action only at the beginning of the episode, not during each step
I am working in an reinforcement learning environment with 1-dimensional action space. My action is only used at the first timestep of an episode and never again. In other word the action only affects ...
3
votes
1
answer
67
views
How to interpret the policy notation $\pi_{\theta}(a_{t}|s_{t})$ in Reinforcement Learning?
In the context of Reinforcement Learning, I have seen that the policy $\pi$ (for some algorithms) is nothing but a Neural Network architecture (for example a Feedforward Neural Network).
This policy ...
- 31
0
votes
0
answers
33
views
Training model for a board game with large actions space
that has a NxM board, and players take turns putting their "dots" on the board a X,Y coordinate, taking up the space. When a few connected dots surround an opponent dot(s), the surrounding ...
- 101
0
votes
1
answer
143
views
Where does the term $\log \mu(u \mid s)$ come from?
This question comes from trying to build a SAC model. The action space is derived from a log normal distribution.
If in the appendix c of the original paper the equation for the log policy is:
$\log \...
- 15
5
votes
2
answers
945
views
What is curriculum learning in reinforcement learning?
I recently came across the term "curriculum learning" in the context of DRL and was intrigued by its potential to improve the learning process. As such, what is curriculum learning? And how ...
- 1,898
2
votes
1
answer
88
views
Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents
Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for ...
- 2,180
2
votes
0
answers
216
views
Choosing and Designing Decay Types for Epsilon-Greedy Exploration in Reinforcement Learning
I am working on a reinforcement learning project that involves epsilon-greedy exploration. I have two questions regarding the choice between linear and exponential decay for epsilon, and the ...
- 41
0
votes
0
answers
14
views
Training two DQNs at the same time: one for task allocation, the other for action selection
i am currently trying to tackle eternity 2 wich is a tile matching puzzle with a DDQN, but it seems to me that using only one DQN won't do the trick. I am currently using a DQN to select an action ...
0
votes
0
answers
50
views
Problem of extremely varied reward value in DDQN
I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here.
...
- 101
0
votes
1
answer
36
views
Are platformer games, with the camera centred on the character, examples of egocentric vision?
An example may be CoinRun, where the character sprite is always centred in the camera view, while the environment moves as a result of player input.
To me, this sounds like egocentric vision, although ...
- 309
0
votes
0
answers
59
views
DQN does not learn the CartPole problem
I've reviewed my code several times ... I can't spot the problem. When I train the agent, it doesn't learn anything. I guess the problem is the way I compute the loss function but I don't know any ...
0
votes
0
answers
28
views
Is there any benefit to using different polyak numbers for target policy and target Q-function parameters?
In the context of RL and DDPG, it seems polyak averaging is used with reference to two things; the target policy, and the target Q parameters. I know that these hyper parameters can be a bit finicky ...
- 145
0
votes
0
answers
52
views
A2C unable to solve Cartpole
I have coded my own A2C implementation using PyTorch. However, despite having followed the algorithm pseudo-code from several sources, my implementation is not able to achieve a proper Cartpole ...
- 26