Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
1 vote
0 answers
12 views

Why slow-changing policy invalidates Double DQN approach in TD3 paper?

In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
Jerry Ding's user avatar
2 votes
0 answers
37 views

Why does only Deep Q Learning have an overestimation bias?

There is a lot of discussion about the overestimation bias for Deep Q Learning and similar off-policy action value estimation algorithms like DDPG. This is why methods like Double DQN and TD3 were ...
Jerry Ding's user avatar
0 votes
0 answers
28 views

How to apply DRL to solve a problem that involves mixed discrete-continuous action spaces where the action's size changes over time?

I have a reinforcement learning problem where a possible action is a probability vector $[p_1\ldots,p_n]$ of size $n\in\{1,\ldots,N\}$, where each element $p_i$ of the vector is between $0$ and $1$ ...
zdm's user avatar
  • 301
1 vote
1 answer
52 views

RL agent for autonomous vehicle is able to follow the road but can't avoid crashing at all (Highway-Env / Racetrack Env.)

I coded some deep RL algorithms (DQN and SAC) with tf2/keras to solve an environment where a vehicle needs to follow the track and avoid crashing into one other vehicle (there is only one other ...
rafiqollective's user avatar
0 votes
0 answers
53 views

Unable to interpret DDPG actor-critic loss curves

I am training a DDPG actor-critic agent and ploting rewards and loss curves each episode to track the training evolution. Rewards values in the plot correspond to the total reward per episode divided ...
davipeix's user avatar
0 votes
1 answer
60 views

Trading bot with RL, automated actions, nonconvergence

I am playing around with RL to develop a trading bot (using DQN). (Disclaimer: I know, that short term stock movements are near-random and having a bot that is actually useful not likely to happen. ...
Andy's user avatar
  • 1
1 vote
1 answer
98 views

Reinforcement Learning vs Supervised Learning [duplicate]

I have never tried reinforcement learning in my life. I'm planning to apply it in robotics. I have some experiences using supervised learning mainly deep learning. So, that's mean I will use neural ...
Muhammad Ikhwan Perwira's user avatar
1 vote
1 answer
54 views

How to deal with infinite loops in the MCTS search of AlphaTensor when using a transposition table?

In the published version of the AlphaTensor algorithm, there are two mentions of a transposition table: In addition, a transposition table is used to recombine different action sequences if they ...
Tristan Nemoz's user avatar
0 votes
1 answer
35 views

Gradient: any resource on how to understand everything about it?

I have read some resources about AI, and they all speak about the gradient. Is there any book focused on this? maybe with tons of images / diagrams? Cheers
zerunio's user avatar
1 vote
2 answers
60 views

How to handle the dead agent in multi-agent environment?

I try to implement deep reinforcement learning on a defender-vs-attacker problem, where agents can be destroyed by enemies. I am coding both the environment and the RL algorithm. The agent can observe ...
zhixin's user avatar
  • 15
0 votes
0 answers
28 views

Deep Reinforcement Learning that takes action from two different sets

I am working on a problem where I want to schedule multiple activities (a1, a2, a3, ... aN) requiring different resource types ...
zeeshan's user avatar
0 votes
1 answer
22 views

Sequential models and distribution shift in RL

We know the problem of "distribution shift" in deep Reinforcement Learning, where the change in policy during training affects the behavior of the agent and therefore the distribution of the ...
SuperTardigrade's user avatar
3 votes
0 answers
51 views

Why policy gradient theorem has two different forms?

I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton &...
Yuxiang Wei's user avatar
0 votes
0 answers
24 views

Policy Gradient in Partial Observability

Let $\pi_{\theta}$ be a policy. Then, I was able to follow through the proof of: $\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|s_i)R(\tau)]$, where ...
A J's user avatar
  • 21
0 votes
1 answer
60 views

Understanding KL Stopping and KL Cutoff for the PPO algorithm

I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the ...
desert_ranger's user avatar
1 vote
0 answers
18 views

How does recurrent neural network implement model based RL system purely in its activation dynamics (in blackbox meta-rl setting)?

I have read these papers "learning to reinforcement learn" and "PFC as meta RL system". The authors claim that when RNN is trained on multiple tasks from a task distribution using ...
veerendra's user avatar
1 vote
1 answer
99 views

What kind of observation state would you give for that environment?

I'm making a new environment where I have two sphere (one above the other) in a 2D plan. I would like some advice on what observation state I should give to my RL. Today I have given the following: ...
CyDevos's user avatar
  • 145
1 vote
1 answer
38 views

why learn an observation model when training latent space model in model based rl

I'm currently studying reinforcement learning through CS 285 provided by UC Berkeley. At 1:52 of the part 5 of the lecture 11, I got confused on why one would want to learn an observation model $p(o_t ...
platoDev's user avatar
0 votes
0 answers
44 views

Using deep reinforcement learning for malware detection; trained agent mostly performs the same action

I'm trying to implement this article: Ransomware early detection using deep reinforcement learning on portable executable header The article uses an unpublished dataset of benign and ransomware ...
soosan123's user avatar
0 votes
0 answers
14 views

Proof of Difference in Return Between Two Policies

I am attempting to understand why Lemma 6.1 holds in this paper on reinforcement learning. I have two questions. First, when defining the value function V(s), why is there a leading (1-γ) term? In the ...
Nikhil Sridhar's user avatar
0 votes
1 answer
106 views

Is my PPO agent learning? or is it just exploring?

I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src. My doubts are mainly due ...
GPra's user avatar
  • 1
0 votes
0 answers
54 views

Question about feature matrix and notation in the paper Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning

Let me know if this is not the place for this question. I'll take it down happily if that's the case. Also, I emailed the authors, but it seems like I won't be getting a response, so that's why I am ...
Schach21's user avatar
  • 242
3 votes
2 answers
554 views

How do you deal with movement inertia in an environment after a step?

I was wondering how can we deal with movement inertia in an environment that is constantly changing? Imagine that you make a step on an environment that moves a ball. When you make the step, you make ...
CyDevos's user avatar
  • 145
0 votes
0 answers
21 views

Changing Gym Environment Mid Training

I am using a custom gym environment for a research project. As my agent solves the environment, I want the task to get progressively harder. Right now, I am doing that like so: ...
user20057611's user avatar
0 votes
0 answers
15 views

Help defining environment with complex action space

I'm working on a personal MARL project with a high-dimensional and continuous action space. The environment is designed to give positive rewards to actions between some moving limits of the action ...
Sebastian Tinoco's user avatar
-1 votes
1 answer
101 views

RL framework to optimize my custom multi-agent simulator [closed]

I have built a custom discrete event simulator with multiple agents and want to optimize the system using RL frameworks that support multi-agent configurations. I will use custom policies. Which ...
RookieScientist's user avatar
0 votes
1 answer
24 views

Can I add additional arguments to my custom Gym Environment? [closed]

I have a custom gym (not gymnasium) environment that I am using for research. I am currently using gym version 0.19.0 installed using conda-forge. Glossing over a lot of details, the agent is learning ...
user20057611's user avatar
0 votes
1 answer
25 views

How does DeepQ learn for different environments?

I'm studying Deep Reinforcement Learning using the book 'DRL in Action' by Zai and Brown. In chapter 3, they present the classic GridWorld example, which can be randomly initialized. This means that ...
Hermes Morales's user avatar
0 votes
1 answer
196 views

How do I add Entropy to a PPO algorithm?

I learned about adding entropy to RL algorithms through the notes provided in SpinningUp. They explained how entropy is added to the SAC algorithm. Here is my understanding - In entropy regularized RL,...
desert_ranger's user avatar
2 votes
1 answer
79 views

What do simulations mean in the context of the AlphaGoZero paper?

I am familiar with the Monte Carlo Tree Search, which can be broken down into 4 parts - Selection, Expansion, Simulation and Backpropogation. In this case, the simulation step denotes performing a ...
desert_ranger's user avatar
1 vote
1 answer
36 views

How to handle large dimensionality differences between state and action inputs in a reinforcement learning predictor?

I'm currently writing code for a reward predictor function r=f(s,a) in reinforcement learning, where 's' is the state with 256 dimensions (the embedding dimension after visual input is processed by an ...
XiaoBanni's user avatar
0 votes
0 answers
13 views

When should grayscale processing be applied to image inputs in visual reinforcement learning environments?

I am currently working with visual environments in Reinforcement Learning (RL) and have noticed differing practices regarding preprocessing of image inputs. Specifically, in the Atari environment, a ...
XiaoBanni's user avatar
2 votes
1 answer
68 views

Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?

I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in ...
XiaoBanni's user avatar
0 votes
0 answers
45 views

How to tell an agent that some actions in the action space are currently not available in gym and the design of action space

I want to make a task allocation decision by reinforcement learning. Suppose there are N tasks to be allocated and M severs to serve these task. However, there is a constraint that one task should be ...
Reese's user avatar
  • 1
1 vote
0 answers
24 views

Is there any Deep RL method that is based on value function approximation of Post-decision States

I am trying to construct an RL algorithm for managing a fleet of vehicles to maximize profit. As far as I know, the Sequential Decision Process can be decomposed into the following pic: My current ...
PokeLu's user avatar
  • 111
1 vote
0 answers
34 views

Deep Q problem. Some q-Values are always greater

Description My problem is to make actor play on stock market. I am trying to teach him playing on some portion of data. I interpolated data to uniformed interals and normalized or standarized all ...
Grzegorz Krug's user avatar
0 votes
0 answers
61 views

What would be an appropriate way of making an AI for Cell Machine?

The game Cell Machine is absolutely by itself not suitable for AI usage, but a certain niche within a mod of the game (Mystic Mod) known as "vaults" does seem like a suitable use case for AI....
user avatar
1 vote
1 answer
101 views

How would one normalize observations in off-policy online reinforcement learning?

In off-policy algorithms such as DQN, you need to feed your input to a network twice. 1. When inputting into a network for predicting the Q values. 2. When feeding the input from the buffer to the ...
desert_ranger's user avatar
1 vote
1 answer
103 views

Fit Q Evaluation in offline reinforcement learning

I am working on a PyTorch implementation of Implicit Q-Learning (IQL) (paper), given a dataset $\mathcal D = \left\{ (\mathbf s_i, \mathbf a_i, \mathbf s_i', r_i ) \right\}$ of transitions. I think I ...
Novice's user avatar
  • 111
2 votes
1 answer
158 views

How does one normalize observations in online reinforcement learning

I was wondering how would one normalize observations to a policy without knowing the upper and lower limits of the environment values. A trivial technique would be normalize each observation by its ...
desert_ranger's user avatar
2 votes
1 answer
206 views

Can I train an agent with DQN, avoiding obstacles and still finding the optimal path to finish the task?

The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible without ...
Mamora's user avatar
  • 53
2 votes
1 answer
139 views

When using (s,a,r,s') to train networks could the Q network be adjusting to a suboptimal r?

My question here is that whenever you take an experience from the experience buffer (s,a,r,s') and you input r into r+yMAXQ(s',a') to get the loss. What if the r from that experience (s,a,r,s') is not ...
Stef's user avatar
  • 53
2 votes
0 answers
21 views

In DQN how does the Q network not converge to the incorrect target? [duplicate]

Whenever you are doing reinforcement learning you periodically update the target network based on the weights of the Q network. While I do understand this helps create a stable target I do not ...
Stef's user avatar
  • 53
0 votes
0 answers
42 views

MADDPG: reward drops fast after updating ac network

I'm using MADDPG to do computation offloading. But the reward drops fast once I start sampling from the replay buffer. ...
hhhhhh's user avatar
  • 1
0 votes
0 answers
31 views

2D layout problem with RL - questions about state representation

I want to solve the following sort of problem with RL, specifically Q-learning (using the tensorflow-agents library): The input is a polygon $\mathcal{P}$ and a list of rectangle sizes $\mathcal{L}=\{...
Felix Goldberg's user avatar
0 votes
0 answers
48 views

DDQN doesn't learn my modified version of Taxi-v3 from OpenAI

I tried to enhance the Taxi-v3 environment from OpenAI (true credit goes of course to https://arxiv.org/abs/cs/9905014) to be capable of handling multiple taxis/passengers. Don't worry, I'm not ...
So S's user avatar
  • 101
3 votes
1 answer
251 views

Take action only at the beginning of the episode, not during each step

I am working in an reinforcement learning environment with 1-dimensional action space. My action is only used at the first timestep of an episode and never again. In other word the action only affects ...
Optical_flow_lover's user avatar
3 votes
1 answer
75 views

How to interpret the policy notation $\pi_{\theta}(a_{t}|s_{t})$ in Reinforcement Learning?

In the context of Reinforcement Learning, I have seen that the policy $\pi$ (for some algorithms) is nothing but a Neural Network architecture (for example a Feedforward Neural Network). This policy ...
moth123's user avatar
  • 31
0 votes
0 answers
34 views

Training model for a board game with large actions space

that has a NxM board, and players take turns putting their "dots" on the board a X,Y coordinate, taking up the space. When a few connected dots surround an opponent dot(s), the surrounding ...
MeLight's user avatar
  • 101
0 votes
1 answer
172 views

Where does the term $\log \mu(u \mid s)$ come from?

This question comes from trying to build a SAC model. The action space is derived from a log normal distribution. If in the appendix c of the original paper the equation for the log policy is: $\log \...
chadmc's user avatar
  • 15

1
2 3 4 5
11