Questions tagged [deep-rl]

For questions related to deep reinforcement learning (DRL), that is, RL combined with deep learning. More precisely, deep neural networks are used to represent e.g. value functions or policies.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

RL - Can RL be applied to problems where the next state is not the next observation?

I'm quite new on the study of reinforcement learning, and Im working on a communication problem with continuous large actions range for my final graduation work. I'm trying to use Gaussian Policy and ...
0 votes
0 answers
8 views

Is model-based RL better suited for domain shift then model-free RL?

My intuition is that richer representations can be used for a larger number of downstream tasks and that model-based RL is more suited to produce such representations. Is there empirical work that ...
1 vote
0 answers
30 views

DDQN Snake keeps crashing into the wall [closed]

Edit: I managed to fix this by changing the optimizer to SGD. I am very new to reinforcement learning, and I attempted to create a DDQN for the game snake but for some reason it keeps learning to ...
0 votes
0 answers
22 views

Embedding layers/entities in openAI's Hide and seek paper

I've recently come across a youtube video about openAI's hide and seek paper (https://openai.com/blog/emergent-tool-use/) and got really fascinated about the paper itself. But as I digging in the ...
0 votes
0 answers
11 views

How can I keep markov property when controlling many agents?

I am working on a project in which I am training a multiagent system to find a minimum in a scalar field. I have many agents that will receive information about the position of some of the other ...
2 votes
1 answer
40 views

Why $V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s,a),\forall s \in S$ in reinforcement learning?

When i read some notes about RL i encounterd following equation and try to prove it: $$ V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s, a),\forall s \in S $$ Here is my attemption: Firstly, i only need to ...
  • 21
2 votes
1 answer
46 views

Propagating gradients through an "Item Selector" network

Consider the following problem: There are $N$ items and $S$ slots. Each item is a vector of length $D$. The goal is to train a neural network to select one item per slot in order to minimize the loss ...
  • 141
1 vote
1 answer
52 views

Neural Network output for the game of Checkers

I'm trying to train a RL agent to play the game of checkers (AlphaZero style) and so far I've managed a proof of concept training a connect 4 agent up until perfection. However, unlike connect 4, ...
2 votes
2 answers
58 views

Why clip the PPO objective on only one side?

In PPO with clipped surrogate objective (see the paper here), we have the following objective: The shape of the function is shown in the image below, and depends on whether the advantage is positive ...
  • 21
0 votes
0 answers
19 views

Action space for a single agent environment with multiple actions

How to define an action space in a gym environment where the agent´s output is a tensor of shape [1,25]? I am working on Travelling salesman problem using DRL where the NN(agent) output`s the sequence ...
1 vote
1 answer
48 views

Deep Reinforcement Learning - How to Avoid a Naive Solution?

I'm implementing a connect-4 agent: using DQN, training by playing vs previous versions of the network. However, many times the network learns that it's best to simply put 4 pieces in the same column, ...
2 votes
0 answers
35 views

Are there Reinforcement Learning algorithms speciallized for the case gamma=0?

I have a Reinforcement Learning problem where the optimal policy does not depend on the next state (ie gamma equals 0). I think this means that I only need an efficient exploration algorithm coupled ...
  • 21
0 votes
0 answers
16 views

Designing a Target Location Environment for DeepRL

I'm trying to make an environment where my agent needs to navigate through a continuous space (using a continuous action space) to get to a target location. Currently, I spawn the agent and the target ...
  • 101
0 votes
0 answers
24 views

Graph problem MDP formulation

I am doing my thesis about trying to learn graph problems using reinforcement learning. I am currently using stablebaselines3 and openAI gym to execute my research but I am having an issue with ...
1 vote
1 answer
17 views

How to normalizing various elements of the reward function?

Suppose I have a reward function $R$ that I wish to penalize w.r.t two distinct phenomenons $A$ and $B$. $A$, for example, could represent the phenomenon of the state not crossing some boundary $[s_1,...
1 vote
0 answers
37 views

Compare Stable-Baselines3 vs. Tianshou

What would you recommend between Stable-Baselines3 and Tianshou for applied research in Reinforcement Learning? Can anyone provide a comparison of the strengths and weaknesses of each library? Or at ...
2 votes
2 answers
65 views

Are batches useful for REINFORCE without strong episode cutoffs?

I'm following along with PyTorch's example implementations (found here) of reinforcement learning algorithms that happen to be largely REINFORCE (vanilla policy gradient) based, and I notice they don'...
  • 89
0 votes
0 answers
147 views

Reinforcement Learning with PPO - entropy loss dropping, but so is performance. Why?

I'm using PPO with an action-mask and I'm encountering a weird phenomenon. At first during training, the entropy loss is decreasing (I interpret this as less exploration, more exploitation, more "...
0 votes
0 answers
12 views

overcoming the original policy with offline RL

I am doing a comparison between RL and a metaheuristic algorithm. What I have found is that online RL does a good work but does not overcome the latter algorithm. In this case generating the samples ...
  • 49
0 votes
0 answers
33 views

Deep reinforcement learning:DQN can't converge

i am work on a project.i use the dqn to maximize return. this picture are some env states. i found that dqn did learn a bit, but after a while it stopped improving and even started to decline. this my ...
0 votes
1 answer
49 views

Why Phasic Policy Gradient (PPG) can update value function in auxiliary phase?

My questions is that how could we train the value network (separated from shared network) by using data from previous policies, which varies a lot since we collect data from different policies with ...
0 votes
0 answers
29 views

How to design rewards in RL?

I am a bit confused regarding rewards in reinforcement learning. In my quite simple environment, where the agent has to find it's way to a target and kill it, the agent has control over heading ...
  • 31
0 votes
1 answer
56 views

Help on Deep Sarsa algorithm that work with pytorch (Adam optimiser) but not with keras/Tensorflow (Adam optimiser)

I have a deep sarsa algorithm wich work great on Pytorch on lunar-lander-v2 and I would use with Keras/Tensorflow. It use mini-batch of size 64 wich are used 128 time to train at each episode. There ...
  • 1
1 vote
1 answer
49 views

How can rewards and loss calculation be extended to multiple agents in a vanilla policy gradient RL setting?

Say I have a simple multi-agent reinforcement learning problem using vanilla policy gradient methods (i.e. REINFORCE) that is currently running with one network per agent. If I can say that each of my ...
  • 89
0 votes
1 answer
59 views

What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?

I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, ...
  • 89
0 votes
1 answer
84 views

How is reinforcement learning applied in the real industry?

I'm a newbie to reinforcement learning. While studying reinforcement learning, a question arose about how to apply reinforcement learning in the real world. Assuming that a reinforcement learning ...
0 votes
0 answers
18 views

What are some approaches for specifying goals for deep-RL agents?

I'm wondering what are the approaches for specifying goals for a trained deep-RL in deployment? E.g. how to tell a car drive agent to go to location $y$? To elaborate, I understand that, for example, ...
1 vote
1 answer
24 views

How to use RL on a robotic moving arm?

I'm working on a simulation of a motor that is attached to a wing (Later, this will also have a real-life counterpart once I'll assemble all the components in our lab), and I can control the forces/...
0 votes
0 answers
29 views

When use positive or negative rewards in reinforcement learning? Is there anything in literature?

Let's say I can design a reward as function of a distance $d>0$ from the target in 2 ways: $r=\frac{1}{1+d}$ or $r=-d$. The first is defined in $(0,1]$ the second in $(-\infty,0]$. I would expect ...
  • 41
0 votes
0 answers
20 views

how to calculate the virtual time of a reinforcement learning model

I have heard about some RL models like alpha go being trained for days but in reality has gained thousands of years of experience and a model which teaches a 3d figure how to walk and fight being ...
1 vote
1 answer
23 views

Why not use only expert demonstrations in Imitation Learning approaches?

Some IL approaches train the agents by using some specific ratio of expert demonstrations to trajectories generated using the policy being optimized. In the specific paper I'm reading they say "...
0 votes
0 answers
22 views

Why does providing an extra prediction output help stabilize training?

I am reading the PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning paper where they tackle the multi-agent path finding problem using reinforcement learning. The problem is ...
1 vote
1 answer
46 views

Can RL still learn if part of my actions are only used once, at the beginning of the episode?

I am working in an environment with 3-dimensional action space. The first two actions are only used at the first timestep and never again. The third action is used at every timestep. Say, the action ...
0 votes
0 answers
43 views

When does it make sense to switch from discrete action space to continuous?

I'm currently working on a custom RL environment for a PPO model that I'd like to have 40-100 discrete actions with integer-level precision (no decimals). Looking through some papers on the topic, it'...
  • 1
0 votes
0 answers
29 views

Double DQN performs worse than DQN

I have an agent that has to explore a customized environment. The environment is a grid (100 squares horizontally, 100 squares vertically, each square is 10 meters wide). In the environment, there are ...
  • 146
0 votes
1 answer
71 views

Which paper describes the effect of learning_starts in Reinforcement Learning?

I have seen many popular RL libraries have a learning_start parameter. This allows the agent to collect enough experiences before training on the replay_buffer. However, I am unable to find the paper ...
0 votes
0 answers
42 views

What does maximal |Q| mean in DQN?

I am reading this paper and came across the term maximal |Q|. I'd like to know whether it refers to the Q values of the current state $Q(s_t,a_t)$ or that of the target $\mathbb{max}_aQ(S_{t+1}, A_t)$....
1 vote
2 answers
41 views

Are RL algorithms suppose to keep learning?

I don't understand if the purposes of RL agents is simply optimizing a model with a reward instead of using labeled data (i.e. in a supervision fashion), or they have also the purpose of keep training ...
  • 41
0 votes
0 answers
130 views

Why DDPG losses don't decrease while the reward grows?

I've noticed that training a DDPG agent in the Reacher-v2 environment of OpenAI Gym, the losses of both actor and critic first decrease but after a while start increasing but the episode mean reward ...
  • 41
1 vote
2 answers
109 views

RL solutions for OpenAI Gym environments?

Is there any place where people share their agent's settings for solving OpenAI Gym Environments? For example, I'd like to know what are good parameters for a DDPG agent to learn the task in Reacher-...
  • 41
0 votes
0 answers
46 views

Is there benefit to autoregressive models for deep RL tasks with long episodes and short required context?

General Case In deep RL (specifically in the space of policy gradient methods) it seems very common that encoder-decoder models (either transformer or RNN-variant) are used in the policy/value ...
  • 101
2 votes
1 answer
367 views

How to deal with small reward values

In my environment rewards are generally small, e.g. [-0.01, 0.01]. My concern is that small reward values might get dominated or distorted by the noise during the training. Does it make sense to scale ...
  • 311
1 vote
0 answers
34 views

What Kind of Reinforcement Learning Algorithms Can Be Used when the Action Space is Unfeasibly Large?

I know Deep Q network as a $S\times A$ DNN which maps the $S$ dimensional statespace to q-values of $A$ distinct actions. In my problem, the action space is still discrete, and finite, but depending ...
  • 111
1 vote
1 answer
81 views

Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?

I understand how to derive the vanilla policy gradient $$ \begin{align} \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t = 0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_{t} \...
  • 35
2 votes
0 answers
35 views

Can objective function and gradient be unlimited in reinforcement learning?

I'm looking at an example where they define a policy $\pi_\theta(a_t|s_t)\sim \mathcal{N}(ks_t, \sigma)$, where $a_t$ and $s_t$ are action and state, while $\theta=(k,\sigma)$ are the parameters of ...
  • 41
1 vote
0 answers
21 views

Why can't I train like a dataset of samples instead of maintaining replay buffer?

On observing the DDPG algorithm, we notice that the updation of neural networks is happening during the episode. But, it seems there is no issue if we allow the completion of an episode and then treat ...
  • 3,381
2 votes
1 answer
76 views

How to sample the tuples during the initial time steps of the DDPG algorithm?

I am facing an issue in understanding the following line from the pseudocode of the DDPG algorithm Sample a random minibatch of $N$ transitions $(s_i, a_i, r_i, s_{i+1})$ from $R$ Here $N$ is a ...
  • 3,381
1 vote
0 answers
26 views

Does deep RL techniques only interested in 'unit transitions' rather than 'whole experience'?

In deep-rl techniques, if I understand correctly, a replay buffer is used in training the neural networks. The purpose of using the replay buffer is to store the experience and send a (sampled) batch ...
  • 3,381
3 votes
1 answer
221 views

Why do Q-values diverge without a target network?

After reviewing similar posts on this topic, I understand that a target network is used to prevent "divergence", but am not sure what it actually means. Q-values are predicted using a ...
2 votes
0 answers
62 views

How does one detect training instabilities in DQN?

I am curious what training instabilities look like in a standard dqn, with or without a target network. I'm assuming the loss function would never converge since the difference between the predicted q-...

1
2 3 4 5
9