Questions tagged [deep-rl]
For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.
532
questions
0
votes
1
answer
21
views
When to stop a DQN agent train?
Hello AI Stack people,
I'm in doubt to when i should stop my DQN agent train.
I'm traning a DQN agent and i will use a hyperparameter optimization method (probabily random search). So, i need a ...
1
vote
1
answer
26
views
PPO learning to achieve a non-Markovian task?
I've been trying to train agents to achieve a non-Markovian task in a modified version of PettingZoo's Waterworld. In the task, I have two pursuers (the agents I'm training) and three evaders. I won't ...
- 11
3
votes
2
answers
352
views
Can reinforcement learning rewards be a combination of current and new state?
I'm structuring the reward function for my RL agent and considering a combination of both the current state and the new state after taking an action. From what I understand, this is possible based on ...
- 139
0
votes
0
answers
9
views
Stable Baselines 3 multiple custom networks in one agent
I'm working with an environment that can easily be subdivided into two parts, with part 1 have an indirect effect on part 2, but I can't simulate either parts alone in a realistic way.
Also, both ...
- 1
1
vote
1
answer
35
views
Do State Variables in RL Models Need Direct Update Equations?
I'm working on a simulation model using RL to optimize an objective function. I'm trying to understand if I need to select my state variables such that I can write state update equations for each one ...
- 139
0
votes
1
answer
61
views
Should I use reinforcement learning to automate the throw of a ball in a Pokemon game?
I am trying to automate the throw of a ball in a game in order to get "Excellent Throw".
To achieve an "Excellent" throw, you need to hit the center of the shrinking target circle ...
- 111
0
votes
0
answers
16
views
Combination of components to maximize a multi-criteria objective function
I have been given a list of components, with various “contributions” (or weights) which put together in a weighted combination have a combined aggregate effect. I then have the task of suggesting ...
- 101
-1
votes
1
answer
54
views
Is the Invalid Action Masking in RL related only to the state or can be dependent on the reward as well?
I'm reading about Invalid Action Masking in RL in order to use it in my PPO algorithm for a specific task.
The problem is that I read such explanations: here, here and here the there the invalid ...
- 204
1
vote
1
answer
34
views
What are the most common methods for handling non-stationary environments in reinforcement learning?
What are the most common algorithms, methods for handling non-stationary environments in reinforcement learning?
- 361
0
votes
0
answers
38
views
Two-agent sequential RL
I have the following RL model that I want to train (see the diagram below). My idea is to have two agents: agent A and agent B. Agent A observes the input I1 and ...
- 309
0
votes
0
answers
29
views
Can a trained RL network outperforms the best training sample?
I'm working on solving a problem where I need to determine the optimal set of actions to find the path that yields the maximum reward. I'm currently using a Deep Q-Network (DQN) for this task. However,...
- 1
0
votes
1
answer
42
views
Why parameter-based RL methods are not widely used?
Parameter-based RL methods perturb parameters of a general function approximator directly, rather than adding noise to the resulting actions. Why they haven't received the same attention and gain ...
- 361
2
votes
1
answer
50
views
Actor Critic need to find the goal to have good update and suceed?
Im trying to solve MoutainCar and CarRacing (i love cars) in gym environnement with DDPG and my algo struggle.
I have think on why this don't work and I would like to know if my resoning is false.
...
1
vote
0
answers
54
views
Handling large or multiple agent actions that cause non-stationarity
How to handle a problem where the agents' actions are changing the environment? For instance, a trading agent can hypothetically make a huge transaction that is enough to push the stock prices to go ...
- 361
2
votes
2
answers
73
views
Learning rate and rewards in Deep Reinforcement Learning
In environments with sparse and (sometimes) binary rewards such as Taxi Cab, Mountain Car and Frozen Lake, where agents rarely encounter positive outcomes, I've observed that the gradient descent's ...
- 116
0
votes
1
answer
49
views
Can all RL algorithms learn with discrete state spaces?
This question come to mind when i was planing to do a benchmark of RL algorithms to my Environment.
In fact, Q-Learning, SARSA actually only handles with discrete state spaces because they are tabular ...
3
votes
1
answer
81
views
Why do big policy updates cause performance drop in deep RL?
In the TRPO and PPO papers, it is mentioned that large policy updates often lead to performance drops in policy gradient methods.
By "large policy updates," they mean a significant KL ...
- 191
0
votes
1
answer
99
views
Is reinforcement learning suitable for application automation?
I have basically automatised the use of an app through the use of OCR and computer vision.
So basically when a word or an image is detected it will perform a certain action.
When that action is ...
- 111
1
vote
0
answers
33
views
Proper way to load a RL (reinforcement learning) model (pytorch) for "testing"...?
I'm working on a RL problem where, in a nutshell, an agent has to go from point A to point B, in that order, with as few steps as possible, using DQN with PyTorch, to train the agent.
During training, ...
2
votes
1
answer
49
views
Should the experience replay memory only contain unique experiences?
I'm training an RL agent/model (DRL/DQN).
Say that, for each learning step, the memory replay used by the agent to learn, has N elements (experiences) stored, where only X are unique elements (...
0
votes
0
answers
34
views
How can I improve these reinforcement learning reward functions?
I am working for the very first time to a RL problem. More in details, I am working with a 3DOF model of a buisness jet aircraft. The environement is made up by the aircraft model + the two PID ...
- 1
3
votes
1
answer
226
views
Can we implement a memory in a REINFORCE algorithm for RL?
In Q-learning with function approximators such as Neural Networks, we typically implement a memory so that at the end of each episode we also train on past experiences. This is typically fine because ...
0
votes
0
answers
24
views
How to design a custom openai gym environment to carry out 5G resource slicing?
PROBLEM AT HAND: I have a resource (Bandwidth) of B Hz. I have to distribute the bandwidth B to users as per their requirements. For instance, voice calls would require some amount of bandwidth while ...
0
votes
0
answers
92
views
What's the action space in RLHF for LLM?
I've been trying to understand how the modern LLMs use PPO for fine-tuning. In the PPO algorithm, one has to compute advantages, which are then used for either increasing or decreasing action's ...
- 191
0
votes
2
answers
39
views
Neural network for specific numbers from a range (Q learning)
PROBLEM AT HAND: I have a resource (Bandwidth) of B Hz. I have to distribute the bandwidth B to users as per their requirements. For instance, voice calls would require some amount of bandwidth while ...
0
votes
1
answer
34
views
How to find an argument of a NN function(which returns a distribution) to minimize a KL divergence?
Consider a neural network function $f:\mathbb{R}\to distribution$. For simplicity, maybe consider that it returns a gaussian distribution.
I want to find $\arg\min_{s\in\mathbb{R}}D_{KL}(f(s),q)$ for ...
- 45
0
votes
0
answers
28
views
Difficulty training PPO agent for robotic arm navigation task
I'm currently working on training a PPO agent for a robotic arm navigation task, where the goal is to navigate the robotic arm to different positions in the environment. I've been training the agent ...
1
vote
0
answers
18
views
How to measure accuracy of learned value function of a fixed policy?
Let's say we've a given policy whose value function is to be evaluated. One way to get the value function can be using expected SARSA, as in this stack exchange answer. However, my MDP's state space ...
- 111
0
votes
1
answer
82
views
Can DQN learn with discrete state spaces?
For example in Cart Pole v1 gym environment the state space is continuous, but we discretize it to apply the Q-Learning algorithm because Q-Learning is a tabular method and only works with discrete ...
0
votes
0
answers
23
views
Policy gradient - future looking returns
In the policy gradient approach, one differentiates the expected reward
$$
\mathbb{E}J=\sum P(\tau;\theta) R(\tau)
$$
to obtain
$$
\Sigma R(\tau) \nabla \log P(\tau;\theta)
$$
(with some abuse of ...
- 1
0
votes
0
answers
17
views
How to clip actions based on state in RL environment for trading (and other tips to approach optimal trade execution)?
I’m currently trying to use RL to analyze price impact in financial markets for optimal trade execution and have coded up a custom gymnasium environment to do so. Now, I'm deciding on which RL ...
- 1
3
votes
1
answer
281
views
Does the DoubleDQN algorithm use a target network or two separate policies?
I've been looking for ways to improve my DQN. That is when I found the Double DQN algorithm. After looking at explanatory videos and posts, I've seen conflicting information:
The Double DQN algorithm ...
0
votes
1
answer
35
views
Can/should a reward function depend on something other than state in a DQN
Question: Is it OK to have a reward function on a DQN or any RL algorithm that depends on variables other than the enviroment state? I'm asking because, so far I'm learning from tutorials, but I've ...
0
votes
1
answer
74
views
Use your own simulation to train a reinforcement learning multi-agent
I am wanting to train an RL multi-agent model to run in a propietary simulation, which is written in C++. Is there a way to change the simulation itself to create an agent, or must I use a ...
0
votes
1
answer
40
views
CNN Input shape for DQN Q-calculating Network
Context: I want to build a DQN with as CNN for calculating its Q value on each step.
Enviroment's status can be described by the attributes of 3 machines (each one with own attributes). I'd also like ...
2
votes
1
answer
103
views
How are POMDPs solved in practice?
In the literature that I've seen so far on how to either exactly or approximately solve POMDPs (Partially-Observable Markov Decision Processes), there seems to be a lot of focus placed on maintaining ...
- 123
0
votes
0
answers
25
views
Reinforcement learning for a word game
Let's imagine a simple word game where there is a grid of letters. The agent starts at a letter and moves to a neighboring letter, continuing in this fashion to form a word. For instance in this grid ...
- 1
0
votes
0
answers
9
views
What is the name of this construction for a compound policy that operates over distinct action sets?
I am developing an RL algorithm with a policy that needs to compute valid probabilities over multiple distinct action sets. I think I have a construction that will work, but I do not know what it is ...
- 1
2
votes
2
answers
58
views
Implementation difference in REINFORCE algorithm, where to sum from
I have a question regarding the implementation of the REINFORCE algorithm.
In berkeley course (see slide 9) the gradient is defined as
Note that the return sums from 1. However in Sutton's book the ...
- 33
0
votes
0
answers
56
views
How do I deal with dynamic, parameterized, action spaces?
I want to design an AI Learning Algorithm for a Student made, round based Game.
Let me first explain the Game/Environment
You have a round based HTTP Game, in which multiple Players can participate.
...
- 1
0
votes
0
answers
26
views
What methods are available for this kind of RL with partially unknown rewards?
Let me give an example. There is a king with 1 million subjects. He wants to maximize the discounted sum of future happiness of these subjects. However, he never fully knows their happiness. He can ...
- 179
0
votes
0
answers
33
views
How to implement neural networks for the Pendulum Swing-Up Environment
I have recently completed the Prediction and Control with Function Approximation course on Coursera, which is part of a reinforcement learning specialisation from the University of Alberta.
One of the ...
- 1
1
vote
1
answer
88
views
How is state space normalization done in off-policy algorithms like dqn? [closed]
There are 4 features in my state representation and they are in different ranges. So I'm thinking state normalization would reduce the bias on certain features. And also, in the problem I consider, ...
- 11
1
vote
0
answers
17
views
Enhancing Generalization in DRL Agents in Static Data Environments
Context: I'm working with a deep reinforcement learning (DRL) agent in a market-like environment where its actions do not affect the environment. The environment uses historical data up to a certain ...
0
votes
1
answer
65
views
Which RL algorithms can be used in an environment where actions have to be performed only in specific situations?
I am wondering which RL algorithms can be used in an environment where actions have to be performed only in specific situations. For example, on a conveyor belt on which a box that fulfills certain ...
- 223
0
votes
0
answers
27
views
Can RL solve scheduling problems with unknown function
I have the following scheduling problem.
There are $n$ tasks and $m>n$ machines.
Each task $i$ has a requirement $t_i$ that should be guaranteed.
Any task can be scheduled on at least one machine ...
- 101
2
votes
2
answers
72
views
Selection of actions based on distribution of rewards
In "A Distributional Perspective on Reinforcement Learning" Bellamare et. al. 2017. We find the following phrase.
As in DQN, we use a simple $\epsilon$-greedy policy over the expected ...
- 1,477
6
votes
2
answers
338
views
DQN arXiv 10-year anniversary: What are the outstanding problems being actively researched in deep Q-learning since 2019?
Background
As of today (12-19-2023), the arXiv submission of the original deep Q-learning approach to achieve superhuman performance on ATARI games has turned a decade old. The original approach, ...
- 1,568
1
vote
0
answers
34
views
Why slow-changing policy invalidates Double DQN approach in TD3 paper?
In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
- 131
2
votes
0
answers
142
views
Why does only Deep Q Learning have an overestimation bias?
There is a lot of discussion about the overestimation bias for Deep Q Learning and similar off-policy action value estimation algorithms like DDPG. This is why methods like Double DQN and TD3 were ...
- 131