Questions tagged [deep-rl]
For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.
503
questions
1
vote
0
answers
12
views
Why slow-changing policy invalidates Double DQN approach in TD3 paper?
In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
- 131
2
votes
0
answers
37
views
Why does only Deep Q Learning have an overestimation bias?
There is a lot of discussion about the overestimation bias for Deep Q Learning and similar off-policy action value estimation algorithms like DDPG. This is why methods like Double DQN and TD3 were ...
- 131
0
votes
0
answers
28
views
How to apply DRL to solve a problem that involves mixed discrete-continuous action spaces where the action's size changes over time?
I have a reinforcement learning problem where a possible action is a probability vector $[p_1\ldots,p_n]$ of size $n\in\{1,\ldots,N\}$, where each element $p_i$ of the vector is between $0$ and $1$ ...
- 301
1
vote
1
answer
52
views
RL agent for autonomous vehicle is able to follow the road but can't avoid crashing at all (Highway-Env / Racetrack Env.)
I coded some deep RL algorithms (DQN and SAC) with tf2/keras to solve an environment where a vehicle needs to follow the track and avoid crashing into one other vehicle (there is only one other ...
0
votes
0
answers
53
views
Unable to interpret DDPG actor-critic loss curves
I am training a DDPG actor-critic agent and ploting rewards and loss curves each episode to track the training evolution. Rewards values in the plot correspond to the total reward per episode divided ...
- 1
0
votes
1
answer
60
views
Trading bot with RL, automated actions, nonconvergence
I am playing around with RL to develop a trading bot (using DQN). (Disclaimer: I know, that short term stock movements are near-random and having a bot that is actually useful not likely to happen. ...
- 1
1
vote
1
answer
98
views
Reinforcement Learning vs Supervised Learning [duplicate]
I have never tried reinforcement learning in my life.
I'm planning to apply it in robotics.
I have some experiences using supervised learning mainly deep learning. So, that's mean I will use neural ...
1
vote
1
answer
54
views
How to deal with infinite loops in the MCTS search of AlphaTensor when using a transposition table?
In the published version of the AlphaTensor algorithm, there are two mentions of a transposition table:
In addition, a transposition table is used to recombine different action sequences if they ...
- 123
0
votes
1
answer
35
views
Gradient: any resource on how to understand everything about it?
I have read some resources about AI, and they all speak about the gradient.
Is there any book focused on this? maybe with tons of images / diagrams?
Cheers
- 3
1
vote
2
answers
60
views
How to handle the dead agent in multi-agent environment?
I try to implement deep reinforcement learning on a defender-vs-attacker problem, where agents can be destroyed by enemies. I am coding both the environment and the RL algorithm. The agent can observe ...
- 15
0
votes
0
answers
28
views
Deep Reinforcement Learning that takes action from two different sets
I am working on a problem where I want to schedule multiple activities (a1, a2, a3, ... aN) requiring different resource types ...
- 1
0
votes
1
answer
22
views
Sequential models and distribution shift in RL
We know the problem of "distribution shift" in deep Reinforcement Learning, where the change in policy during training affects the behavior of the agent and therefore the distribution of the ...
3
votes
0
answers
51
views
Why policy gradient theorem has two different forms?
I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton &...
- 31
0
votes
0
answers
24
views
Policy Gradient in Partial Observability
Let $\pi_{\theta}$ be a policy. Then, I was able to follow through the proof of: $\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|s_i)R(\tau)]$, where ...
- 21
0
votes
1
answer
60
views
Understanding KL Stopping and KL Cutoff for the PPO algorithm
I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the ...
- 608
1
vote
0
answers
18
views
How does recurrent neural network implement model based RL system purely in its activation dynamics (in blackbox meta-rl setting)?
I have read these papers "learning to reinforcement learn" and "PFC as meta RL system". The authors claim that when RNN is trained on multiple tasks from a task distribution using ...
- 19
1
vote
1
answer
99
views
What kind of observation state would you give for that environment?
I'm making a new environment where I have two sphere (one above the other) in a 2D plan.
I would like some advice on what observation state I should give to my RL.
Today I have given the following:
...
- 145
1
vote
1
answer
38
views
why learn an observation model when training latent space model in model based rl
I'm currently studying reinforcement learning through CS 285 provided by UC Berkeley.
At 1:52 of the part 5 of the lecture 11, I got confused on why one would want to learn an observation model $p(o_t ...
- 13
0
votes
0
answers
44
views
Using deep reinforcement learning for malware detection; trained agent mostly performs the same action
I'm trying to implement this article:
Ransomware early detection using deep reinforcement learning on portable executable header
The article uses an unpublished dataset of benign and ransomware ...
0
votes
0
answers
14
views
Proof of Difference in Return Between Two Policies
I am attempting to understand why Lemma 6.1 holds in this paper on reinforcement learning. I have two questions. First, when defining the value function V(s), why is there a leading (1-γ) term? In the ...
0
votes
1
answer
106
views
Is my PPO agent learning? or is it just exploring?
I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src.
My doubts are mainly due ...
- 1
0
votes
0
answers
54
views
Question about feature matrix and notation in the paper Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
Let me know if this is not the place for this question. I'll take it down happily if that's the case. Also, I emailed the authors, but it seems like I won't be getting a response, so that's why I am ...
- 242
3
votes
2
answers
554
views
How do you deal with movement inertia in an environment after a step?
I was wondering how can we deal with movement inertia in an environment that is constantly changing?
Imagine that you make a step on an environment that moves a ball. When you make the step, you make ...
- 145
0
votes
0
answers
21
views
Changing Gym Environment Mid Training
I am using a custom gym environment for a research project. As my agent solves the environment, I want the task to get progressively harder. Right now, I am doing that like so:
...
0
votes
0
answers
15
views
Help defining environment with complex action space
I'm working on a personal MARL project with a high-dimensional and continuous action space. The environment is designed to give positive rewards to actions between some moving limits of the action ...
-1
votes
1
answer
101
views
RL framework to optimize my custom multi-agent simulator [closed]
I have built a custom discrete event simulator with multiple agents and want to optimize the system using RL frameworks that support multi-agent configurations. I will use custom policies. Which ...
0
votes
1
answer
24
views
Can I add additional arguments to my custom Gym Environment? [closed]
I have a custom gym (not gymnasium) environment that I am using for research. I am currently using gym version 0.19.0 installed using conda-forge. Glossing over a lot of details, the agent is learning ...
0
votes
1
answer
25
views
How does DeepQ learn for different environments?
I'm studying Deep Reinforcement Learning using the book 'DRL in Action' by Zai and Brown. In chapter 3, they present the classic GridWorld example, which can be randomly initialized. This means that ...
- 221
0
votes
1
answer
196
views
How do I add Entropy to a PPO algorithm?
I learned about adding entropy to RL algorithms through the notes provided in SpinningUp. They explained how entropy is added to the SAC algorithm. Here is my understanding - In entropy regularized RL,...
- 608
2
votes
1
answer
79
views
What do simulations mean in the context of the AlphaGoZero paper?
I am familiar with the Monte Carlo Tree Search, which can be broken down into 4 parts - Selection, Expansion, Simulation and Backpropogation. In this case, the simulation step denotes performing a ...
- 608
1
vote
1
answer
36
views
How to handle large dimensionality differences between state and action inputs in a reinforcement learning predictor?
I'm currently writing code for a reward predictor function r=f(s,a) in reinforcement learning, where 's' is the state with 256 dimensions (the embedding dimension after visual input is processed by an ...
- 51
0
votes
0
answers
13
views
When should grayscale processing be applied to image inputs in visual reinforcement learning environments?
I am currently working with visual environments in Reinforcement Learning (RL) and have noticed differing practices regarding preprocessing of image inputs. Specifically, in the Atari environment, a ...
- 51
2
votes
1
answer
68
views
Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?
I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in ...
- 51
0
votes
0
answers
45
views
How to tell an agent that some actions in the action space are currently not available in gym and the design of action space
I want to make a task allocation decision by reinforcement learning. Suppose there are N tasks to be allocated and M severs to serve these task.
However, there is a constraint that one task should be ...
- 1
1
vote
0
answers
24
views
Is there any Deep RL method that is based on value function approximation of Post-decision States
I am trying to construct an RL algorithm for managing a fleet of vehicles to maximize profit. As far as I know, the Sequential Decision Process can be decomposed into the following pic:
My current ...
- 111
1
vote
0
answers
34
views
Deep Q problem. Some q-Values are always greater
Description
My problem is to make actor play on stock market. I am trying to teach him playing on some portion of data. I interpolated data to uniformed interals and normalized or standarized all ...
0
votes
0
answers
61
views
What would be an appropriate way of making an AI for Cell Machine?
The game Cell Machine is absolutely by itself not suitable for AI usage, but a certain niche within a mod of the game (Mystic Mod) known as "vaults" does seem like a suitable use case for AI....
user74135
1
vote
1
answer
101
views
How would one normalize observations in off-policy online reinforcement learning?
In off-policy algorithms such as DQN, you need to feed your input to a network twice. 1. When inputting into a network for predicting the Q values. 2. When feeding the input from the buffer to the ...
- 608
1
vote
1
answer
103
views
Fit Q Evaluation in offline reinforcement learning
I am working on a PyTorch implementation of Implicit Q-Learning (IQL) (paper), given a dataset $\mathcal D = \left\{ (\mathbf s_i, \mathbf a_i, \mathbf s_i', r_i ) \right\}$ of transitions. I think I ...
- 111
2
votes
1
answer
158
views
How does one normalize observations in online reinforcement learning
I was wondering how would one normalize observations to a policy without knowing the upper and lower limits of the environment values. A trivial technique would be normalize each observation by its ...
- 608
2
votes
1
answer
206
views
Can I train an agent with DQN, avoiding obstacles and still finding the optimal path to finish the task?
The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible without ...
- 53
2
votes
1
answer
139
views
When using (s,a,r,s') to train networks could the Q network be adjusting to a suboptimal r?
My question here is that whenever you take an experience from the experience buffer (s,a,r,s') and you input r into r+yMAXQ(s',a') to get the loss. What if the r from that experience (s,a,r,s') is not ...
- 53
2
votes
0
answers
21
views
In DQN how does the Q network not converge to the incorrect target? [duplicate]
Whenever you are doing reinforcement learning you periodically update the target network based on the weights of the Q network. While I do understand this helps create a stable target I do not ...
- 53
0
votes
0
answers
42
views
MADDPG: reward drops fast after updating ac network
I'm using MADDPG to do computation offloading. But the reward drops fast once I start sampling from the replay buffer.
...
- 1
0
votes
0
answers
31
views
2D layout problem with RL - questions about state representation
I want to solve the following sort of problem with RL, specifically Q-learning (using the tensorflow-agents library):
The input is a polygon $\mathcal{P}$ and a list of rectangle sizes $\mathcal{L}=\{...
- 129
0
votes
0
answers
48
views
DDQN doesn't learn my modified version of Taxi-v3 from OpenAI
I tried to enhance the Taxi-v3 environment from OpenAI (true credit goes of course to https://arxiv.org/abs/cs/9905014) to be capable of handling multiple taxis/passengers. Don't worry, I'm not ...
- 101
3
votes
1
answer
251
views
Take action only at the beginning of the episode, not during each step
I am working in an reinforcement learning environment with 1-dimensional action space. My action is only used at the first timestep of an episode and never again. In other word the action only affects ...
3
votes
1
answer
75
views
How to interpret the policy notation $\pi_{\theta}(a_{t}|s_{t})$ in Reinforcement Learning?
In the context of Reinforcement Learning, I have seen that the policy $\pi$ (for some algorithms) is nothing but a Neural Network architecture (for example a Feedforward Neural Network).
This policy ...
- 31
0
votes
0
answers
34
views
Training model for a board game with large actions space
that has a NxM board, and players take turns putting their "dots" on the board a X,Y coordinate, taking up the space. When a few connected dots surround an opponent dot(s), the surrounding ...
- 101
0
votes
1
answer
172
views
Where does the term $\log \mu(u \mid s)$ come from?
This question comes from trying to build a SAC model. The action space is derived from a log normal distribution.
If in the appendix c of the original paper the equation for the log policy is:
$\log \...
- 15