Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
1 vote
2 answers
26 views

How to handle the dead agent in multi-agent environment?

I recently try to implement deep reinforcement learning on a defender-vs-attacker problem, where agents can be destroyed by enemies. I am coding both the environment and the RL algorithm. The agent ...
zhixin's user avatar
  • 11
0 votes
0 answers
21 views

Deep Reinforcement Learning that takes action from two different sets

I am working on a problem where I want to schedule multiple activities (a1, a2, a3, ... aN) requiring different resource types ...
zeeshan's user avatar
0 votes
1 answer
19 views

Sequential models and distribution shift in RL

We know the problem of "distribution shift" in deep Reinforcement Learning, where the change in policy during training affects the behavior of the agent and therefore the distribution of the ...
SuperTardigrade's user avatar
1 vote
0 answers
21 views

Why policy gradient theorem has two different forms?

I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton &...
Yuxiang Wei's user avatar
0 votes
0 answers
19 views

Policy Gradient in Partial Observability

Let $\pi_{\theta}$ be a policy. Then, I was able to follow through the proof of: $\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|s_i)R(\tau)]$, where ...
A J's user avatar
  • 1
0 votes
1 answer
35 views

Understanding KL Stopping and KL Cutoff for the PPO algorithm

I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the ...
desert_ranger's user avatar
1 vote
0 answers
13 views

How does recurrent neural network implement model based RL system purely in its activation dynamics (in blackbox meta-rl setting)?

I have read these papers "learning to reinforcement learn" and "PFC as meta RL system". The authors claim that when RNN is trained on multiple tasks from a task distribution using ...
veerendra's user avatar
1 vote
1 answer
92 views

What kind of observation state would you give for that environment?

I'm making a new environment where I have two sphere (one above the other) in a 2D plan. I would like some advice on what observation state I should give to my RL. Today I have given the following: ...
CyDevos's user avatar
  • 135
1 vote
1 answer
35 views

why learn an observation model when training latent space model in model based rl

I'm currently studying reinforcement learning through CS 285 provided by UC Berkeley. At 1:52 of the part 5 of the lecture 11, I got confused on why one would want to learn an observation model $p(o_t ...
platoDev's user avatar
0 votes
0 answers
41 views

Using deep reinforcement learning for malware detection; trained agent mostly performs the same action

I'm trying to implement this article: Ransomware early detection using deep reinforcement learning on portable executable header The article uses an unpublished dataset of benign and ransomware ...
soosan123's user avatar
0 votes
0 answers
13 views

Proof of Difference in Return Between Two Policies

I am attempting to understand why Lemma 6.1 holds in this paper on reinforcement learning. I have two questions. First, when defining the value function V(s), why is there a leading (1-γ) term? In the ...
Nikhil Sridhar's user avatar
0 votes
1 answer
74 views

Is my PPO agent learning? or is it just exploring?

I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src. My doubts are mainly due ...
GPra's user avatar
  • 1
0 votes
0 answers
54 views

Question about feature matrix and notation in the paper Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning

Let me know if this is not the place for this question. I'll take it down happily if that's the case. Also, I emailed the authors, but it seems like I won't be getting a response, so that's why I am ...
Schach21's user avatar
  • 242
2 votes
2 answers
530 views

How do you deal with movement inertia in an environment after a step?

I was wondering how can we deal with movement inertia in an environment that is constantly changing? Imagine that you make a step on an environment that moves a ball. When you make the step, you make ...
CyDevos's user avatar
  • 135
0 votes
0 answers
19 views

Changing Gym Environment Mid Training

I am using a custom gym environment for a research project. As my agent solves the environment, I want the task to get progressively harder. Right now, I am doing that like so: ...
user20057611's user avatar
0 votes
0 answers
12 views

Help defining environment with complex action space

I'm working on a personal MARL project with a high-dimensional and continuous action space. The environment is designed to give positive rewards to actions between some moving limits of the action ...
Sebastian Tinoco's user avatar
-1 votes
1 answer
39 views

RL framework to optimize my custom multi-agent simulator [closed]

I have built a custom discrete event simulator with multiple agents and want to optimize the system using RL frameworks that support multi-agent configurations. I will use custom policies. Which ...
RookieScientist's user avatar
0 votes
1 answer
23 views

Can I add additional arguments to my custom Gym Environment? [closed]

I have a custom gym (not gymnasium) environment that I am using for research. I am currently using gym version 0.19.0 installed using conda-forge. Glossing over a lot of details, the agent is learning ...
user20057611's user avatar
0 votes
1 answer
22 views

How does DeepQ learn for different environments?

I'm studying Deep Reinforcement Learning using the book 'DRL in Action' by Zai and Brown. In chapter 3, they present the classic GridWorld example, which can be randomly initialized. This means that ...
Hermes Morales's user avatar
0 votes
1 answer
106 views

How do I add Entropy to a PPO algorithm?

I learned about adding entropy to RL algorithms through the notes provided in SpinningUp. They explained how entropy is added to the SAC algorithm. Here is my understanding - In entropy regularized RL,...
desert_ranger's user avatar
1 vote
1 answer
74 views

What do simulations mean in the context of the AlphaGoZero paper?

I am familiar with the Monte Carlo Tree Search, which can be broken down into 4 parts - Selection, Expansion, Simulation and Backpropogation. In this case, the simulation step denotes performing a ...
desert_ranger's user avatar
1 vote
1 answer
26 views

How to handle large dimensionality differences between state and action inputs in a reinforcement learning predictor?

I'm currently writing code for a reward predictor function r=f(s,a) in reinforcement learning, where 's' is the state with 256 dimensions (the embedding dimension after visual input is processed by an ...
XiaoBanni's user avatar
0 votes
0 answers
10 views

When should grayscale processing be applied to image inputs in visual reinforcement learning environments?

I am currently working with visual environments in Reinforcement Learning (RL) and have noticed differing practices regarding preprocessing of image inputs. Specifically, in the Atari environment, a ...
XiaoBanni's user avatar
1 vote
1 answer
47 views

Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?

I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in ...
XiaoBanni's user avatar
0 votes
0 answers
31 views

How to tell an agent that some actions in the action space are currently not available in gym and the design of action space

I want to make a task allocation decision by reinforcement learning. Suppose there are N tasks to be allocated and M severs to serve these task. However, there is a constraint that one task should be ...
Reese's user avatar
  • 1
1 vote
0 answers
23 views

Is there any Deep RL method that is based on value function approximation of Post-decision States

I am trying to construct an RL algorithm for managing a fleet of vehicles to maximize profit. As far as I know, the Sequential Decision Process can be decomposed into the following pic: My current ...
PokeLu's user avatar
  • 111
0 votes
0 answers
32 views

Deep Q problem. Some q-Values are always greater

Description My problem is to make actor play on stock market. I am trying to teach him playing on some portion of data. I interpolated data to uniformed interals and normalized or standarized all ...
Grzegorz Krug's user avatar
0 votes
0 answers
60 views

What would be an appropriate way of making an AI for Cell Machine?

The game Cell Machine is absolutely by itself not suitable for AI usage, but a certain niche within a mod of the game (Mystic Mod) known as "vaults" does seem like a suitable use case for AI....
user avatar
1 vote
1 answer
87 views

How would one normalize observations in off-policy online reinforcement learning?

In off-policy algorithms such as DQN, you need to feed your input to a network twice. 1. When inputting into a network for predicting the Q values. 2. When feeding the input from the buffer to the ...
desert_ranger's user avatar
0 votes
1 answer
27 views

Fit Q Evaluation in offline reinforcement learning

I am working on a PyTorch implementation of Implicit Q-Learning (IQL) (paper), given a dataset $\mathcal D = \left\{ (\mathbf s_i, \mathbf a_i, \mathbf s_i', r_i ) \right\}$ of transitions. I think I ...
Novice's user avatar
  • 101
1 vote
1 answer
128 views

How does one normalize observations in online reinforcement learning

I was wondering how would one normalize observations to a policy without knowing the upper and lower limits of the environment values. A trivial technique would be normalize each observation by its ...
desert_ranger's user avatar
2 votes
1 answer
187 views

Can I train an agent with DQN, avoiding obstacles and still finding the optimal path to finish the task?

The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible without ...
Mamora's user avatar
  • 53
2 votes
1 answer
138 views

When using (s,a,r,s') to train networks could the Q network be adjusting to a suboptimal r?

My question here is that whenever you take an experience from the experience buffer (s,a,r,s') and you input r into r+yMAXQ(s',a') to get the loss. What if the r from that experience (s,a,r,s') is not ...
Stef's user avatar
  • 53
2 votes
0 answers
20 views

In DQN how does the Q network not converge to the incorrect target? [duplicate]

Whenever you are doing reinforcement learning you periodically update the target network based on the weights of the Q network. While I do understand this helps create a stable target I do not ...
Stef's user avatar
  • 53
0 votes
0 answers
29 views

MADDPG: reward drops fast after updating ac network

I'm using MADDPG to do computation offloading. But the reward drops fast once I start sampling from the replay buffer. ...
hhhhhh's user avatar
  • 1
0 votes
0 answers
29 views

2D layout problem with RL - questions about state representation

I want to solve the following sort of problem with RL, specifically Q-learning (using the tensorflow-agents library): The input is a polygon $\mathcal{P}$ and a list of rectangle sizes $\mathcal{L}=\{...
Felix Goldberg's user avatar
0 votes
0 answers
27 views

DDQN doesn't learn my modified version of Taxi-v3 from OpenAI

I tried to enhance the Taxi-v3 environment from OpenAI (true credit goes of course to https://arxiv.org/abs/cs/9905014) to be capable of handling multiple taxis/passengers. Don't worry, I'm not ...
So S's user avatar
  • 101
3 votes
1 answer
235 views

Take action only at the beginning of the episode, not during each step

I am working in an reinforcement learning environment with 1-dimensional action space. My action is only used at the first timestep of an episode and never again. In other word the action only affects ...
Optical_flow_lover's user avatar
3 votes
1 answer
67 views

How to interpret the policy notation $\pi_{\theta}(a_{t}|s_{t})$ in Reinforcement Learning?

In the context of Reinforcement Learning, I have seen that the policy $\pi$ (for some algorithms) is nothing but a Neural Network architecture (for example a Feedforward Neural Network). This policy ...
moth123's user avatar
  • 31
0 votes
0 answers
33 views

Training model for a board game with large actions space

that has a NxM board, and players take turns putting their "dots" on the board a X,Y coordinate, taking up the space. When a few connected dots surround an opponent dot(s), the surrounding ...
MeLight's user avatar
  • 101
0 votes
1 answer
143 views

Where does the term $\log \mu(u \mid s)$ come from?

This question comes from trying to build a SAC model. The action space is derived from a log normal distribution. If in the appendix c of the original paper the equation for the log policy is: $\log \...
chadmc's user avatar
  • 15
5 votes
2 answers
945 views

What is curriculum learning in reinforcement learning?

I recently came across the term "curriculum learning" in the context of DRL and was intrigued by its potential to improve the learning process. As such, what is curriculum learning? And how ...
Robin van Hoorn's user avatar
2 votes
1 answer
88 views

Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents

Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for ...
Luca Anzalone's user avatar
2 votes
0 answers
216 views

Choosing and Designing Decay Types for Epsilon-Greedy Exploration in Reinforcement Learning

I am working on a reinforcement learning project that involves epsilon-greedy exploration. I have two questions regarding the choice between linear and exponential decay for epsilon, and the ...
XiaoBanni's user avatar
0 votes
0 answers
14 views

Training two DQNs at the same time: one for task allocation, the other for action selection

i am currently trying to tackle eternity 2 wich is a tile matching puzzle with a DDQN, but it seems to me that using only one DQN won't do the trick. I am currently using a DQN to select an action ...
mt-clemente's user avatar
0 votes
0 answers
50 views

Problem of extremely varied reward value in DDQN

I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here. ...
Subhajit Saha's user avatar
0 votes
1 answer
36 views

Are platformer games, with the camera centred on the character, examples of egocentric vision?

An example may be CoinRun, where the character sprite is always centred in the camera view, while the environment moves as a result of player input. To me, this sounds like egocentric vision, although ...
thesofakillers's user avatar
0 votes
0 answers
59 views

DQN does not learn the CartPole problem

I've reviewed my code several times ... I can't spot the problem. When I train the agent, it doesn't learn anything. I guess the problem is the way I compute the loss function but I don't know any ...
milad heidari's user avatar
0 votes
0 answers
28 views

Is there any benefit to using different polyak numbers for target policy and target Q-function parameters?

In the context of RL and DDPG, it seems polyak averaging is used with reference to two things; the target policy, and the target Q parameters. I know that these hyper parameters can be a bit finicky ...
Justin T's user avatar
  • 145
0 votes
0 answers
52 views

A2C unable to solve Cartpole

I have coded my own A2C implementation using PyTorch. However, despite having followed the algorithm pseudo-code from several sources, my implementation is not able to achieve a proper Cartpole ...
Antonio's user avatar
  • 26

1
2 3 4 5
10