Skip to main content

Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

How to find an argument of a NN function(which returns a distribution) to minimize a KL divergence?

Consider a neural network function $f:\mathbb{R}\to distribution$. For simplicity, maybe consider that it returns a gaussian distribution. I want to find $\arg\min_{s\in\mathbb{R}}D_{KL}(f(s),q)$ for ...
user3315463's user avatar
0 votes
0 answers
16 views

Difficulty training PPO agent for robotic arm navigation task

 I'm currently working on training a PPO agent for a robotic arm navigation task, where the goal is to navigate the robotic arm to different positions in the environment. I've been training the agent ...
Weitao Kang's user avatar
1 vote
0 answers
12 views

How to measure accuracy of learned value function of a fixed policy?

Let's say we've a given policy whose value function is to be evaluated. One way to get the value function can be using expected SARSA, as in this stack exchange answer. However, my MDP's state space ...
ModCon's user avatar
  • 111
0 votes
2 answers
41 views

Can DQN learn with discrete state spaces?

For example in Cart Pole v1 gym environment the state space is continuous, but we discretize it to apply the Q-Learning algorithm because Q-Learning is a tabular method and only works with discrete ...
Vitor Martins's user avatar
0 votes
0 answers
22 views

Policy gradient - future looking returns

In the policy gradient approach, one differentiates the expected reward $$ \mathbb{E}J=\sum P(\tau;\theta) R(\tau) $$ to obtain $$ \Sigma R(\tau) \nabla \log P(\tau;\theta) $$ (with some abuse of ...
Eli's user avatar
  • 1
0 votes
0 answers
10 views

How to clip actions based on state in RL environment for trading (and other tips to approach optimal trade execution)?

I’m currently trying to use RL to analyze price impact in financial markets for optimal trade execution and have coded up a custom gymnasium environment to do so. Now, I'm deciding on which RL ...
walrus's user avatar
  • 1
3 votes
1 answer
170 views

Does the DoubleDQN algorithm use a target network or two separate policies?

I've been looking for ways to improve my DQN. That is when I found the Double DQN algorithm. After looking at explanatory videos and posts, I've seen conflicting information: The Double DQN algorithm ...
Vladislav Korecký's user avatar
0 votes
1 answer
25 views

Can/should a reward function depend on something other than state in a DQN

Question: Is it OK to have a reward function on a DQN or any RL algorithm that depends on variables other than the enviroment state? I'm asking because, so far I'm learning from tutorials, but I've ...
Oliver Mohr Bonometti's user avatar
0 votes
1 answer
32 views

Use your own simulation to train a reinforcement learning multi-agent

I am wanting to train an RL multi-agent model to run in a propietary simulation, which is written in C++. Is there a way to change the simulation itself to create an agent, or must I use a ...
michael-c-michael's user avatar
0 votes
1 answer
36 views

CNN Input shape for DQN Q-calculating Network

Context: I want to build a DQN with as CNN for calculating its Q value on each step. Enviroment's status can be described by the attributes of 3 machines (each one with own attributes). I'd also like ...
Oliver Mohr Bonometti's user avatar
2 votes
1 answer
94 views

How are POMDPs solved in practice?

In the literature that I've seen so far on how to either exactly or approximately solve POMDPs (Partially-Observable Markov Decision Processes), there seems to be a lot of focus placed on maintaining ...
QMath's user avatar
  • 123
0 votes
0 answers
20 views

Reinforcement learning for a word game

Let's imagine a simple word game where there is a grid of letters. The agent starts at a letter and moves to a neighboring letter, continuing in this fashion to form a word. For instance in this grid ...
mustafa's user avatar
0 votes
0 answers
7 views

What is the name of this construction for a compound policy that operates over distinct action sets?

I am developing an RL algorithm with a policy that needs to compute valid probabilities over multiple distinct action sets. I think I have a construction that will work, but I do not know what it is ...
Wowee's user avatar
  • 1
2 votes
2 answers
49 views

Implementation difference in REINFORCE algorithm, where to sum from

I have a question regarding the implementation of the REINFORCE algorithm. In berkeley course (see slide 9) the gradient is defined as Note that the return sums from 1. However in Sutton's book the ...
Chris XU's user avatar
0 votes
0 answers
49 views

How do I deal with dynamic, parameterized, action spaces?

I want to design an AI Learning Algorithm for a Student made, round based Game. Let me first explain the Game/Environment You have a round based HTTP Game, in which multiple Players can participate. ...
Andre's user avatar
  • 1
0 votes
0 answers
23 views

What methods are available for this kind of RL with partially unknown rewards?

Let me give an example. There is a king with 1 million subjects. He wants to maximize the discounted sum of future happiness of these subjects. However, he never fully knows their happiness. He can ...
causative's user avatar
  • 149
0 votes
0 answers
26 views

How to implement neural networks for the Pendulum Swing-Up Environment

I have recently completed the Prediction and Control with Function Approximation course on Coursera, which is part of a reinforcement learning specialisation from the University of Alberta. One of the ...
dML's user avatar
  • 1
1 vote
1 answer
63 views

How is state space normalization done in off-policy algorithms like dqn? [closed]

There are 4 features in my state representation and they are in different ranges. So I'm thinking state normalization would reduce the bias on certain features. And also, in the problem I consider, ...
Aki A's user avatar
  • 11
1 vote
0 answers
16 views

Enhancing Generalization in DRL Agents in Static Data Environments

Context: I'm working with a deep reinforcement learning (DRL) agent in a market-like environment where its actions do not affect the environment. The environment uses historical data up to a certain ...
ElonMuskofBadIdeas's user avatar
0 votes
1 answer
62 views

Which RL algorithms can be used in an environment where actions have to be performed only in specific situations?

I am wondering which RL algorithms can be used in an environment where actions have to be performed only in specific situations. For example, on a conveyor belt on which a box that fulfills certain ...
Martin S's user avatar
  • 213
0 votes
0 answers
25 views

Can RL solve scheduling problems with unknown function

I have the following scheduling problem. There are $n$ tasks and $m>n$ machines. Each task $i$ has a requirement $t_i$ that should be guaranteed. Any task can be scheduled on at least one machine ...
Jika's user avatar
  • 101
2 votes
2 answers
61 views

Selection of actions based on distribution of rewards

In "A Distributional Perspective on Reinforcement Learning" Bellamare et. al. 2017. We find the following phrase. As in DQN, we use a simple $\epsilon$-greedy policy over the expected ...
foreverska's user avatar
  • 1,065
5 votes
2 answers
283 views

DQN arXiv 10-year anniversary: What are the outstanding problems being actively researched in deep Q-learning since 2019?

Background As of today (12-19-2023), the arXiv submission of the original deep Q-learning approach to achieve superhuman performance on ATARI games has turned a decade old. The original approach, ...
DeepQZero's user avatar
  • 1,379
1 vote
0 answers
30 views

Why slow-changing policy invalidates Double DQN approach in TD3 paper?

In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
Jerry Ding's user avatar
2 votes
0 answers
83 views

Why does only Deep Q Learning have an overestimation bias?

There is a lot of discussion about the overestimation bias for Deep Q Learning and similar off-policy action value estimation algorithms like DDPG. This is why methods like Double DQN and TD3 were ...
Jerry Ding's user avatar
0 votes
0 answers
38 views

How to apply DRL to solve a problem that involves mixed discrete-continuous action spaces where the action's size changes over time?

I have a reinforcement learning problem where a possible action is a probability vector $[p_1\ldots,p_n]$ of size $n\in\{1,\ldots,N\}$, where each element $p_i$ of the vector is between $0$ and $1$ ...
zdm's user avatar
  • 301
1 vote
1 answer
75 views

RL agent for autonomous vehicle is able to follow the road but can't avoid crashing at all (Highway-Env / Racetrack Env.)

I coded some deep RL algorithms (DQN and SAC) with tf2/keras to solve an environment where a vehicle needs to follow the track and avoid crashing into one other vehicle (there is only one other ...
rafiqollective's user avatar
0 votes
0 answers
179 views

Unable to interpret DDPG actor-critic loss curves

I am training a DDPG actor-critic agent and ploting rewards and loss curves each episode to track the training evolution. Rewards values in the plot correspond to the total reward per episode divided ...
davipeix's user avatar
0 votes
1 answer
158 views

Trading bot with RL, automated actions, nonconvergence

I am playing around with RL to develop a trading bot (using DQN). (Disclaimer: I know, that short term stock movements are near-random and having a bot that is actually useful not likely to happen. ...
Andy's user avatar
  • 1
1 vote
1 answer
154 views

Reinforcement Learning vs Supervised Learning [duplicate]

I have never tried reinforcement learning in my life. I'm planning to apply it in robotics. I have some experiences using supervised learning mainly deep learning. So, that's mean I will use neural ...
Muhammad Ikhwan Perwira's user avatar
1 vote
1 answer
76 views

How to deal with infinite loops in the MCTS search of AlphaTensor when using a transposition table?

In the published version of the AlphaTensor algorithm, there are two mentions of a transposition table: In addition, a transposition table is used to recombine different action sequences if they ...
Tristan Nemoz's user avatar
0 votes
1 answer
48 views

Gradient: any resource on how to understand everything about it?

I have read some resources about AI, and they all speak about the gradient. Is there any book focused on this? maybe with tons of images / diagrams? Cheers
zerunio's user avatar
1 vote
2 answers
86 views

How to handle the dead agent in multi-agent environment?

I try to implement deep reinforcement learning on a defender-vs-attacker problem, where agents can be destroyed by enemies. I am coding both the environment and the RL algorithm. The agent can observe ...
zhixin's user avatar
  • 43
0 votes
0 answers
32 views

Deep Reinforcement Learning that takes action from two different sets

I am working on a problem where I want to schedule multiple activities (a1, a2, a3, ... aN) requiring different resource types ...
zeeshan's user avatar
0 votes
1 answer
31 views

Sequential models and distribution shift in RL

We know the problem of "distribution shift" in deep Reinforcement Learning, where the change in policy during training affects the behavior of the agent and therefore the distribution of the ...
SuperTardigrade's user avatar
4 votes
0 answers
98 views

Why policy gradient theorem has two different forms?

I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton &...
Yuxiang Wei's user avatar
0 votes
0 answers
25 views

Policy Gradient in Partial Observability

Let $\pi_{\theta}$ be a policy. Then, I was able to follow through the proof of: $\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|s_i)R(\tau)]$, where ...
A J's user avatar
  • 21
0 votes
1 answer
196 views

Understanding KL Stopping and KL Cutoff for the PPO algorithm

I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the ...
desert_ranger's user avatar
1 vote
0 answers
27 views

How does recurrent neural network implement model based RL system purely in its activation dynamics (in blackbox meta-rl setting)?

I have read these papers "learning to reinforcement learn" and "PFC as meta RL system". The authors claim that when RNN is trained on multiple tasks from a task distribution using ...
veerendra's user avatar
1 vote
1 answer
106 views

What kind of observation state would you give for that environment?

I'm making a new environment where I have two sphere (one above the other) in a 2D plan. I would like some advice on what observation state I should give to my RL. Today I have given the following: ...
CyDevos's user avatar
  • 145
1 vote
1 answer
56 views

why learn an observation model when training latent space model in model based rl

I'm currently studying reinforcement learning through CS 285 provided by UC Berkeley. At 1:52 of the part 5 of the lecture 11, I got confused on why one would want to learn an observation model $p(o_t ...
platoDev's user avatar
0 votes
0 answers
46 views

Using deep reinforcement learning for malware detection; trained agent mostly performs the same action

I'm trying to implement this article: Ransomware early detection using deep reinforcement learning on portable executable header The article uses an unpublished dataset of benign and ransomware ...
soosan123's user avatar
0 votes
0 answers
17 views

Proof of Difference in Return Between Two Policies

I am attempting to understand why Lemma 6.1 holds in this paper on reinforcement learning. I have two questions. First, when defining the value function V(s), why is there a leading (1-γ) term? In the ...
Nikhil Sridhar's user avatar
0 votes
1 answer
240 views

Is my PPO agent learning? or is it just exploring?

I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src. My doubts are mainly due ...
GPra's user avatar
  • 1
0 votes
0 answers
57 views

Question about feature matrix and notation in the paper Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning

Let me know if this is not the place for this question. I'll take it down happily if that's the case. Also, I emailed the authors, but it seems like I won't be getting a response, so that's why I am ...
Schach21's user avatar
  • 242
3 votes
2 answers
569 views

How do you deal with movement inertia in an environment after a step?

I was wondering how can we deal with movement inertia in an environment that is constantly changing? Imagine that you make a step on an environment that moves a ball. When you make the step, you make ...
CyDevos's user avatar
  • 145
0 votes
0 answers
27 views

Changing Gym Environment Mid Training

I am using a custom gym environment for a research project. As my agent solves the environment, I want the task to get progressively harder. Right now, I am doing that like so: ...
user20057611's user avatar
0 votes
0 answers
28 views

Help defining environment with complex action space

I'm working on a personal MARL project with a high-dimensional and continuous action space. The environment is designed to give positive rewards to actions between some moving limits of the action ...
Sebastian Tinoco's user avatar
-1 votes
1 answer
191 views

RL framework to optimize my custom multi-agent simulator [closed]

I have built a custom discrete event simulator with multiple agents and want to optimize the system using RL frameworks that support multi-agent configurations. I will use custom policies. Which ...
RookieScientist's user avatar
0 votes
1 answer
26 views

Can I add additional arguments to my custom Gym Environment? [closed]

I have a custom gym (not gymnasium) environment that I am using for research. I am currently using gym version 0.19.0 installed using conda-forge. Glossing over a lot of details, the agent is learning ...
user20057611's user avatar

1
2 3 4 5
11