Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

A2C value function outputs keep increasing

I was implementing the A2C algorithm with as close to baseline setup as possible, and this is the code I came up with. The problem is that even after multiple rechecks, the program isn't showing ...
0
votes
1answer
28 views

How would you shape a reward function if there was four quantities to optimize?

I found this article quite useful on how to shape a reward function in RL. However, the example they gave is quite simple, where the goal is to minimize only two quantities (velocity and distance). ...
0
votes
0answers
13 views

Prioritized Experience Replay, clarifications for Important Sampling

I can't seem to understand how the weight equation is dissected and how it really works when combined with the TD-error value. The weight equation is: I can understand what N, P(i) and beta represent,...
1
vote
1answer
11 views

How to enforce action bounds between 0 & 1 in soft actor-critic algorithm?

In the paper "Soft Actor-Critic Algorithms and Applications", appendix C shows enforcing action bounds using the tanh squashing function which is in (-1, 1). I have action bounds in (0, 1), ...
1
vote
1answer
37 views

How to detect entities in Montezuma's Revenge environment

I'm thinking of implementing "Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation" paper. In this paper authors used some custom object ...
-1
votes
0answers
40 views

ADVANTAGE ACTOR CRITIC WITH TWO ACTIONS

What is the Loss Of Advantage Actor Critic in case there are two actions taken place SIMULTANEOUSLY. Is it ok to be written in this way ? please if anyone knows references addressing this matter, that ...
0
votes
0answers
10 views

How do I quantify the difference in sample efficiency for two almost similar methods?

I am comparing my coded TD3 (Twin-Delayed DDPG) and the same TD3 (same hyperparameters) but with Priority Replay Buffer instead of a normal Replay Buffer. From what I have read, PER (Priority ...
0
votes
0answers
29 views

Bridging the gap between simulation and real-world scenarios!

I've got a DRL model that was trained on a simulation at a frame rate of 100fps, after testing it with 100fps it gives good results however when testing it with another frame rate say 50fps it gives a ...
0
votes
1answer
24 views

How to test the robustness of an agent in a custom reinforcement learning environment?

I have used the stable-baseline3 implementation of the SAC algorithm to train policies in a custom gym environment. So far the results look promising. However, I would like to test the robustness of ...
4
votes
1answer
79 views

Deep Q-Learning “catastrophic drop” reasons?

I am implementing some "classical" papers in Model Free RL like DQN, Double DQN, and Double DQN with Prioritized Replay. Through the various models im running on ...
0
votes
1answer
32 views

How to formulate discounted return in cartpole?

I am trying to formulate a problem that aims to prolong the lifetime of the simulation, the same as the Cartpole problem. I aware that there are two types of return: finite horizon undiscounted ...
0
votes
0answers
36 views

Optimal episode length in reinforcement learning

I have a custom environment for stock trading where an episode can be as long as 2000-3000 steps. I've run several experiments with td3 and sac algorithms, average reward per episode flattens after ...
3
votes
2answers
83 views

What are the best hyper-parameters to tune in reinforcement learning?

Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, ...
1
vote
1answer
39 views

Why did Distributional Q Learning go out of popularity?

I read some papers (for example, this) and blogs that spoke about the advantages of distributional Q learning. However, it no longer seems to come up in literature. Did it have any shortcomings that ...
0
votes
0answers
16 views

DQN learns to always choose the same action for all states

I have created an RL model that uses QBased policy with a neural network for estimating Q values. My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. ...
0
votes
0answers
33 views

CartPoleV0 model is not getting trained in even after 1500+ episodes using deep Q-learning

I am new to deep Q learning and trying to train the open AI cartpole_V0 game using deep Q learning. Here is my code: ...
2
votes
1answer
31 views

How does sharing parameters between the policy and value functions help in PPO?

The PPO objective may include a value function error term when parameters are shared between the policy and value functions. How does this help, and when to use a neural network architecture that ...
4
votes
1answer
112 views

What is the effect of parallel environments in reinforcement learning?

Do parallel environments improve the agent's ability to learn or does it not really make a difference? Specifically, I am using PPO, but I think this applies across the board to other algorithms too.
0
votes
0answers
26 views

What is the relationship between buffer size and actor loss in DDPG?

Actor loss in the DDPG algorithm is: critic_value = critic_model([state_batch, action_batch]) actor_loss = -tf.math.reduce_mean(critic_value) As I was trying to ...
1
vote
1answer
56 views

What happens with policy gradient methods if rewards are differentiable?

I would like some help with understanding why there is no explicit flow of information from the reward gradient to the parameters of the policy in policy gradient methods. What I mean is the following,...
2
votes
1answer
31 views

Is there a multi-agent deep reinforcement learning algorithm which is for environments with only discrete action spaces (Not hybrid)?

Is there a multi-agent deep reinforcement learning algorithm which is for environments with only discrete action spaces (Not hybrid) and have centralized training? I have been looking for algorithms, (...
1
vote
1answer
31 views

Understanding Generalized Advantage Estimate in reinforcement learning

I was reading the paper on Generalized Advantage Estimate. It first introduces a generalized form of policy gradient equation without involving $\gamma$ and then it says the following: We will ...
2
votes
0answers
81 views

Update Rule with Deep Q-Learning (DQN) for 2-player games

I am wondering how to correctly implement the DQN algorithm for two-player games such as Tic Tac Toe and Connect 4. While my algorithm is mastering Tic Tac Toe relatively quickly, I cannot get great ...
1
vote
0answers
20 views

Is there any thumb rule on the cardinality of state space in order to use the parameterized function to estimate value functions?

Value functions for a given MDP can be learned in at least two ways by experience. The first way (tabular calculation) is generally used in the case of state spaces that are small enough. The second ...
4
votes
1answer
83 views

What are the major differences between multi-armed bandits and the other well-known algorithms (DQN, A3C, PPO, etc)?

I have studied in the past different algorithms, i.e. DQN, DDQN, REINFORCE, A3C, PPO, TRPO, so on. I am doing an internship this summer where I have to use a multi-armed bandit (MAB). I am a bit ...
0
votes
0answers
10 views

DQN + HER, TD Error spiked then success rate plummets. What went wrong?

TL;DR: I trained a DQN + HER model using stable-baselines library for a custom environment. I noticed that in most runs, sometimes the TD-Error will spike and then the success rate of my model ...
0
votes
1answer
33 views

What would happen to an agent trained using Markov Decision Process if the goal node changes?

I was reading up a paper that did routing based on an MDP, and I was wondering because, in routing, there is a sender node and a receiver node, so if the receiver node changes (sending a message to ...
3
votes
0answers
91 views

How to deal with a moving target in the Lunar Lander environment with DDPG?

I have noticed that DDPG does rather well at solving environments with a static target. For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
1
vote
1answer
31 views

PPO agent for vehicle control does not learn to stop at traffic lights

I have built a custom RL environment with gym, which simulates the RL vehicle and potential vehicles in front of the RL vehicle as well as traffic lights (including ...
3
votes
1answer
79 views

In DQN, would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?

In the classical examples of deep q-learning, I often see neural networks in which the input represents the state of the agent, while the output is a tuple with all the values of $Q(s, a)$ predicted ...
0
votes
0answers
33 views

Should the actor and critic share a common feature extraction neural network?

In an environment with image observations, if we use an actor-critic method to find a good policy, commonly, we will use a feature extraction neural network, such as ResNet, to extract the information ...
-1
votes
1answer
25 views

Using states (features) and actions from a heuristic model to estimate the value function of a reinforcement learning agent [closed]

new to RL here. As far as i understood from RL courses, that there is two sides of reinforcement learning. Policy Evaluation, which is the task of knowing the value function for certain policy. and ...
0
votes
0answers
37 views

PyTorch: LSTM error while trying to update the hidden state

I am trying to train an LSTM while keeping its hidden state (LSTM stateful) until the moment when I am going to start a new epoch(episode). But here it's come an interesting situation because I am ...
2
votes
2answers
239 views

How to fight with unstability in self play?

I'm working on a neural network that plays some board games like reversi or tic-tac-toe (zero-sum games, two players). I'm trying to have one network topology for all the games - I specifically don't ...
2
votes
1answer
57 views

How to recover the target Q network's weights solely from the snapshots of the primary Q network's weights in DQN?

Suppose that I have a DQN agent, which has two neural networks: one is the primary Q network and the other is the target Q network. In every update, the target Q network is updated with a soft update ...
0
votes
1answer
42 views

Is the policy gradient expression in Fundamentals of Deep Learning wrong?

I don't understand the policy gradient as explained in Chapter-9 (Deep Reinforcement Learning) of the book Fundamentals of deep learning. Here is the whole paragraph: Policy Learning via Policy ...
0
votes
0answers
33 views

Backpropagation in REINFORCE algorithms with Categorical / Multinomial Distribution

From a paper by Williams, I know in general how to backpropagate log-probabilities of chosen actions when applying the REINFORCE weight update rule. However, I was wondering about a case not being ...
6
votes
2answers
104 views

Are there RL algorithms that also try to predict the next state?

So far I've developed simple RL algorithms, like Deep Q-Learning and Double Deep Q-Learning. Also, I read a bit about A3C and policy gradient but superficially. If I remember correctly, all these ...
0
votes
1answer
90 views

How to simplify policy gradient theoram to $E_{\pi}[G_t \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]$?

In "Introduction to Reinforcement Learning" (Richard Sutton) section 13.3(Reinforce algorithm) they have the following equation: \begin{align} \nabla_{\theta}J &\propto \sum_s \mu(s) \...
0
votes
0answers
25 views

Deployment of a DeepRL model trained on a custom OpenAI-GYM environment

I developed a custom OpenAI-GYM environment and trained a CDQN model on it, now I am trying to figure out how can I test it not using my gym environment but in production (using real world ...
0
votes
0answers
22 views

How to constrain some actions in a multi-dimensional action space?

In portfolio management (allocation) the action space is given by the weights of the assets, i.e. $\sum_{i=1}^m a_i=1$. There may be some weight constraints like one cannot allocate more than 10% of ...
0
votes
1answer
73 views

Reinforcement Learning for an environment that is non-markovian

I am a beginner in the field of Reinforcement Learning with only a couple of months of experience being in the field. Soon, I will start working on a project where we want to optimize the production ...
0
votes
0answers
16 views

How to have a DNN output a classification for each user at once?

I have a Reinforcement Learning environment with an agent that allocates power values to different users. To do so, I have thought of implementing a deep neural network like the one shown in the ...
0
votes
0answers
47 views

How to scale the action in a custom environment with DDPG?

I am trying to implement DDPG in a custom gym environment. The action is the relative allocation of funds between each asset. The action space is a Box with the ...
1
vote
1answer
34 views

Why is the behaviour policy denoted by $\beta$ and the exploration policy by $ \mu'$ in the DDPG paper?

I am learning about the deep deterministic policy gradient (DDPG) (Lillicrap et al, 2016) and got confused about the notation of the behavior policy. Lillicrap et al. denote the policy gradient by $$\...
0
votes
0answers
7 views

What is the rationale behind the minimap of MAgent?

The MAgent family of PettingZoo is based on a previous implementation that gives a little tutorial explaining the gridworld ...
2
votes
0answers
26 views

What is the name of this algorithm that estimates the gradient with an average by sampling from a distribution?

Consider maximizing the function $R(w)$ with parameter $w$ using gradient ascent. However, we don't know the gradient $\nabla_wR(w)$ formula. Now suppose $w$ is sampled from a probability distribution ...
0
votes
0answers
30 views

Proving existence or non existence of reward function to make given policy “uniquely” optimal when reward function is dependent only on S or both S,A

I was going through paper titled "Algorithms for Inverse Reinforcement Learning" by Andrew Ng and Russell. It states following basics: MDP $M$ is a tuple $(S,A,\{P_{sa}\},\gamma,R)$, where ...
1
vote
1answer
223 views

How do I get started with multi-agent reinforcement learning?

Is there any tutorial that walks through a multi-agent reinforcement learning implementation (in Python) using libraries such as OpenAI's Gym (for the environment), TF-agents, and stable-baselines-3? ...
2
votes
1answer
46 views

How can I compress the states of a reinforcement learning agent?

I'm working on a problem that involves an RL agent with very large states. These states consist of several pieces of information about the agent. The states are not images, so techniques like ...

1
2 3 4 5 6