Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
787
questions with no upvoted or accepted answers
8
votes
0
answers
263
views
Is the Bellman equation that uses sampling weighted by the Q values (instead of max) a contraction?
It is proved that the Bellman update is a contraction (1).
Here is the Bellman update that is used for Q-Learning:
$$Q_{t+1}(s, a) = Q_{t}(s, a) + \alpha*(r(s, a, s') + \gamma \max_{a^*} (Q_{t}(s',
...
- 81
8
votes
0
answers
118
views
Normalizing Normal Distributions in Thompson Sampling for online Reinforcement Learning
In my implementation of Thompson Sampling (TS) for online Reinforcement Learning, my distribution for selecting $a$ is $\mathcal{N}(Q(s, a), \frac{1}{C(s,a)+1})$, where $C(s,a)$ is the number of times ...
- 81
8
votes
0
answers
1k
views
Is there a difference in the architecture of deep reinforcement learning when multiple actions are performed instead of a single action?
I've built a deep deterministic policy gradient reinforcement learning agent to be able to handle any games/tasks that have only one action. However, the agent seems to fail horribly when there are ...
- 433
6
votes
0
answers
98
views
Proof that there always exists a dominating policy in an MDP
I think that it is common knowledge that for any infinite horizon discounted MDP $(S, A, P, r, \gamma)$, there always exists a dominating policy $\pi$, i.e. a policy $\pi$ such that for all policies $\...
- 185
6
votes
0
answers
351
views
How to correctly implement self-play with DQN?
I have an environment where an agent faces an equal opponent, and while I've achieved OK performance implementing DQN and treating the opponent as a part of the environment, I think performance would ...
- 61
6
votes
0
answers
166
views
How exactly does self-play work, and how does it relate to MCTS?
I am working towards using RL to create an AI for a two-player, hidden-information, a turn-based board game. I have just finished David Silver's RL course and Denny Britz's coding exercises, and so am ...
- 61
6
votes
1
answer
3k
views
How to deal with different actions for different states of the environment?
I'm new to this AI/Machine Learning and was playing around with OpenAI Gym a bit. When looking through the environments, I came across the Blackjack-v0 environment, ...
5
votes
0
answers
69
views
What exactly is non-delusional Q-learning?
Problems occur when we combine Q-learning with a function approximator.
What exactly is the delusional-bias and non-delusional Q-learning? I am talking about the neurIPS 18 best paper Non-delusional Q-...
- 183
5
votes
1
answer
221
views
How does the optimization process in hindsight experience replay exactly work?
I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the ...
- 101
5
votes
1
answer
425
views
What could be the cause of the drop in the reward in A3C?
The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause.
The problem has a sparse reward, batch size=2000, <...
- 173
5
votes
1
answer
185
views
How define a reward function for a humanoid agent whose goal is to stand up from the ground?
I'm trying to teach a humanoid agent how to stand up after falling. The episode starts with the agent lying on the floor with its back touching the ground, and its goal is to stand up in the shortest ...
- 151
4
votes
1
answer
371
views
Why is it recommended to use a "separate test environment" when evaluating a model?
I am training an agent (stable baselines3 algorithm) on a custom environment. During training, I want to have a callback so that for every $N$ steps of the learning process, I get the current model ...
4
votes
0
answers
155
views
What is the correct interpretation of the discount factor in MDPs?
In infinite-horizon MDPs one can consider the expected discounted return from a distribution of start states as the objective[^1]. i.e. $\mathbb{E}[V^{\pi}(S_0)] = \mathbb{E}[G_0] = \mathbb{E}[\sum_{t=...
- 41
4
votes
1
answer
199
views
How does OpenAI-ES use Adam?
I just read that OpenAI's ES uses Adam: "OpenAI’s ES is denoted as “OptimES” (since it uses Adam optimizer)"?? I verified they are correct using the link they posted, (see es_distributed/...
- 370
4
votes
0
answers
113
views
What is the Thompson Sampling in simple terms?
I am looking at the different existing methods of action selection in reinforcement learning.
I found several methods like epsilon-greedy, softmax, upper confidence bound and Thompson sampling.
I ...
- 41
4
votes
0
answers
2k
views
Optimal episode length in reinforcement learning
I have a custom environment for stock trading where an episode can be as long as 2000-3000 steps. I've run several experiments with td3 and sac algorithms, average reward per episode flattens after ...
- 341
4
votes
1
answer
265
views
How can I compress the states of a reinforcement learning agent?
I'm working on a problem that involves an RL agent with very large states. These states consist of several pieces of information about the agent. The states are not images, so techniques like ...
- 123
4
votes
0
answers
99
views
Where does this variation of the importance sampling weight come from?
I have seeing a variation in importance sampling (IS) in Prioritized Experience Replay (PER) in some implementations regarding the original paper approach stated as (in section 3.4):
$$
w_{i}=\left(\...
- 81
4
votes
0
answers
135
views
How to update the observation probabilities in a POMDP?
How can I update the observation probability for a POMDP (or HMM), in order to have a more accurate prediction model?
The POMDP relies on observation probabilities that match an observation to a state....
- 85
4
votes
0
answers
79
views
Difficulty in agent's learning with increasing dimensions of continuous actions
I have been working on some RL project, where the policy is controlling the robot using its joint angles.Throughout the project I have noticed some phenomenon, which caught my attention. I have ...
- 41
4
votes
1
answer
98
views
How can I sample the output distribution multiple times when pruning the filters with reinforcement learning?
I was reading the paper Learning to Prune Filters in Convolutional Neural Networks, which is about pruning the CNN filters using reinforcement learning (policy gradient). The paper says that the input ...
- 51
4
votes
0
answers
196
views
Are there reinforcement learning algorithms not based on Markov decision processes?
Are all RL algorithms based on the MDP? If not, could you give examples of some which aren't? I've looked elsewhere, but I haven't seen it explicitly said.
- 613
4
votes
0
answers
65
views
How to deal with nonstationary rewards in asymmetric self-play reinforcement learning?
Suppose we're training two agents to play an asymmetric game from scratch using self play (like Zerg vs. Protoss in Starcraft). During training one of the agents can become stronger (discover a good ...
- 141
4
votes
0
answers
79
views
Unable to train Coach for Banana-v0 Gym environment
I have just started playing with Reinforcement learning and starting from the basics I'm trying to figure out how to solve Banana Gym with coach.
Essentially ...
- 141
4
votes
1
answer
200
views
Why is my implementation of REINFORCE algorithm for portfolio optimization not converging?
I'm trying to implement the Reinforce algorithm (Monte Carlo policy gradient) in order to optimize a portfolio of 94 stocks on a daily basis (I have suitable historical data to achieve this). The idea ...
- 229
4
votes
0
answers
353
views
Is there a way to do reinforcement learning in POMDP?
Are there any algorithms to use reinforcement learning to learn optimal policies in partially observable Markov decision process (POMDP) i.e. when the state is not perfectly observed? More ...
4
votes
0
answers
384
views
What is the simplest policy gradient method to implement for a problem continuous action space?
I have a problem I would like to tackle with RL, but I am not sure if it is even doable.
My agent has to figure out how to fill a very large vector (let's say from 600 to 4000 in the most complex ...
- 145
4
votes
0
answers
924
views
What could be causing the drastic performance drop of the DQN model on the Pong environment?
I am running a basic DQN (Deep Q-Network) on the Pong environment. Not a CNN, just a 3 layer linear neural net with ReLUs.
It seems to work for the most part, but at some point, my model suffers from ...
- 141
4
votes
0
answers
351
views
When using hashing in tile coding, why are memory requirements reduced and there is only a little loss of performance?
In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto explain, on page 221, a form of tile coding using hashing, to reduce memory consumption.
I have two questions ...
- 311
4
votes
0
answers
94
views
Why is there an inconsistency in the definitions of the retrace?
In Section 4.3 of the paper Learning by Playing - Solving Sparse Reward Tasks from Scratch, the authors define Retrace as
$$
Q^{ret}=\sum_{j=i}^\infty\left(\gamma^{j-i}\prod_{k=i}^jc_k\right)[r(s_j,...
- 451
4
votes
0
answers
71
views
What research has been done on learning non-Markovian reward functions?
Recently, some work has been done planning and learning in Non-Markovian Decision Processes, that is, decision-making with temporally extended rewards. In these settings, a particular reward is ...
- 41
4
votes
0
answers
62
views
Does everyone still use discount rates?
In Section 10.4 of Sutton and Barto's RL book, they argue that the discount rate $\gamma$ has no effect in continuing settings. They show (at least for one objective function) that the average of the ...
- 2,038
4
votes
0
answers
89
views
Can deep successor representations be used with the A3C algorithm?
Deep Successor Representations(DSR) has given better performance in tasks like navigation, when compared to normal model-free RL tasks. Basically, DSR is a hybrid of model-free RL and model-based RL. ...
4
votes
0
answers
440
views
RL to generate sentences
I want to develop a system to generate grammatically correct sentences. The input would be some words. The output would be a grammatically correct human-like sentence.
Eg:
Input: capital, Paris, ...
- 161
4
votes
1
answer
805
views
Traveling salesman problem variant: which algorithm to choose?
I have an industrial problem which I'm trying to cast as a Traveling Salesman problem (TSP) in 3D euclidian space. There are physical limitations which implies that some subpaths may or may not be ...
- 41
3
votes
0
answers
26
views
Algorithms for average reward reinforcement learning in continuous/general state-action space
I see that discounted reward reinforcement learning has been extensively studied in the literature. However, the average reward metric receives less attention, and it looks like algorithms for this ...
- 55
3
votes
0
answers
51
views
Why policy gradient theorem has two different forms?
I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton &...
- 31
3
votes
0
answers
197
views
Let's Verify Step by Step: Old wine in new bottles?
In their paper "Let's Verify Step by Step" OpenAI proudly presents a new way of reward learning which shall foster LLMs' capabilities of mathematical and logical reasoning:
We've trained a ...
3
votes
0
answers
76
views
How can i tinker my neural network to learn stronger on rare events?
I am training a neural network on a regression problem.
Most of the time the actual y (label) has the same value (say ~0.2) and only in rare cases the actual y is very different (say 2.0 or -2.0)
...
- 31
3
votes
0
answers
97
views
Are there Reinforcement Learning algorithms specialized for the case $\gamma=0$?
I have a Reinforcement Learning problem where the optimal policy does not depend on the next state (ie gamma equals 0). I think this means that I only need an efficient exploration algorithm coupled ...
- 31
3
votes
0
answers
207
views
How to deal with variable action ranges in RL for continuous action spaces
I am reading this paper on battery management using RL. The action consist in the charging/discharging power of the battery at timestep $t$. For instance, in the case of the charging power, the ...
- 69
3
votes
0
answers
54
views
How to fix high variance of the returns on a 2d env?
I'm trying to train an agent on a self-written 2d env, and it just doesn't converge to the solution.
It is basically a 2d game where you have to move a small circle around the screen and try to avoid ...
- 31
3
votes
0
answers
97
views
How to prove Lemma 1.6 in the book "Reinforcement Learning: Theory and Algorithms"
I am trying to prove the following lemma from Reinforcement Learning: Theory and Algorithms on page 8.
Lemma 1.6. We have that:
$$
\left[(1-\gamma)\left(I-\gamma P^{\pi}\right)^{-1}\right]_{(s, a),\...
- 31
3
votes
0
answers
67
views
How does the paper implement NEAT without a global set tracking Innovations?
I have been reading this paper on NEAT and trying to implement the algorithm in C#. For the most part, I understand everything in the paper however, there are 2 things I don't understand that confuse ...
- 51
3
votes
0
answers
363
views
How to deal with a moving target in the Lunar Lander environment with DDPG?
I have noticed that DDPG does rather well at solving environments with a static target.
For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
- 131
3
votes
0
answers
296
views
How do I implement the cross-entropy-method for a RL environment with a continuous action space?
I found many tutorials and posts on how to solve RL environments with discrete action spaces using the cross entropy method (e.g., in this blog post for the OpenAI Gym frozen lake environment).
...
- 143
3
votes
0
answers
237
views
How to implement REINFORCE with eligibility traces?
The pseudocode below is taken from Barto and Sutton's "Reinforcement Learning: an introduction". It shows an actor-critic implementation with eligibility traces. My question is: if I set $\...
3
votes
0
answers
136
views
Is better to reward short- or long-term progress in Q-learning?
I have been training some kind of agent to reach a target using a Q-learning based approach, and I have tried two different types of rewards:
Long-term reward: $\mathrm{reward} = - \mathrm{distance}(\...
- 1,197
3
votes
0
answers
288
views
Policy gradient: Does it use the Markov property?
To derive the policy gradient, we start by writing the equation for the probability of a certain trajectory (e.g. see spinningup tutorial):
$$
\begin{align}
P_\theta(\tau) &= P_\theta(s_0, a_0, ...
- 131
3
votes
0
answers
443
views
Stack of Planes as the Action Space Representation for AlphaZero (Chess)
I have a question regarding the action space of the policy network used in AlphaZero.
From the paper:
We represent the policy π(a|s) by a 8 × 8 × 73 stack of planes encoding a probability ...
- 137