Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

660 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7
votes
0answers
917 views

Is there a difference in the architecture of deep reinforcement learning when multiple actions are performed instead of a single action?

I've built a deep deterministic policy gradient reinforcement learning agent to be able to handle any games/tasks that have only one action. However, the agent seems to fail horribly when there are ...
6
votes
0answers
175 views

Is the Bellman equation that uses sampling weighted by the Q values (instead of max) a contraction?

It is proved that the Bellman update is a contraction (1). Here is the Bellman update that is used for Q-Learning: $$Q_{t+1}(s, a) = Q_{t}(s, a) + \alpha*(r(s, a, s') + \gamma \max_{a^*} (Q_{t}(s', ...
6
votes
0answers
64 views

Normalizing Normal Distributions in Thompson Sampling for online Reinforcement Learning

In my implementation of Thompson sampling (TS) for online Reinforcement Learning, my distribution for selecting $a$ is $\mathcal{N}(Q(s, a), \frac{1}{C(s,a)+1})$ where $C(s,a)$ is the number of times $...
5
votes
0answers
52 views

Proof that there always exists a dominating policy in an MDP

I think that it is common knowledge that for any infinite horizon discounted MDP $(S, A, P, r, \gamma)$, there always exists a dominating policy $\pi$, i.e. a policy $\pi$ such that for all policies $\...
5
votes
1answer
325 views

Multi Armed Bandits with large number of arms

I'm dealing with a (stochastic) Multi Armed Bandit (MAB) with a large number of arms. Consider a pizza machine that produces a pizza depending on an input $i$ (equivalent to an arm). The (finite) set ...
5
votes
1answer
137 views

What are the state-of-the-art results in OpenAI's gym environments?

What are the state-of-the-art results in OpenAI's gym environments? Is there a link to a paper/article that describes them and how these SOTA results were calculated?
4
votes
0answers
500 views

Why isn't a target network used for the critic in on-policy actor-critic methods?

Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow: $$ V_\pi(s_t)...
4
votes
0answers
75 views

How to update the observation probabilities in a POMDP?

How can I update the observation probability for a POMDP (or HMM), in order to have a more accurate prediction model? The POMDP relies on observation probabilities that match an observation to a state....
4
votes
0answers
65 views

Difficulty in agent's learning with increasing dimensions of continuous actions

I have been working on some RL project, where the policy is controlling the robot using its joint angles.Throughout the project I have noticed some phenomenon, which caught my attention. I have ...
4
votes
1answer
112 views

How does the optimization process in hindsight experience replay exactly work?

I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the ...
4
votes
0answers
70 views

Unable to train Coach for Banana-v0 Gym environment

I have just started playing with Reinforcement learning and starting from the basics I'm trying to figure out how to solve Banana Gym with coach. Essentially ...
4
votes
1answer
139 views

What could be the cause of the drop in the reward in A3C?

The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause. The problem has a sparse reward, batch size=2000, <...
4
votes
0answers
72 views

What is the current state-of-the-art for inverse reinforcement learning on robotic control tasks?

The question is broad, since I am looking for any kind of insight into the subject. I am currently looking into the possibility for useing inverse reinforcement learning for learning robotic ...
4
votes
1answer
106 views

REINFORCE algorithm for portfolio optimization - problem while training

I'm trying to implement the Reinforce algorithm (Monte Carlo policy gradient) in order to optimize a portfolio of 94 stocks on a daily basis (I have suitable historical data to achieve this). The idea ...
4
votes
0answers
166 views

Is there a way to do reinforcement learning in POMDP?

Are there any algorithms to use reinforcement learning to learn optimal policies in partially observable Markov decision process (POMDP) i.e. when the state is not perfectly observed. More ...
4
votes
0answers
299 views

Policy gradient methods for continuous action space

I have a problem I would like to tackle with RL but I am not sure if it is even doable. My agent has to figure out how to fill a very large vector (let's say from 600 to 4000 in the most complex ...
4
votes
0answers
463 views

What could be causing the drastic performance drop of the DQN model on the Pong environment?

I am running a basic DQN (Deep Q-Network) on the Pong environment. Not a CNN, just a 3 layer linear neural net with ReLUs. It seems to work for the most part, but at some point, my model suffers from ...
4
votes
0answers
59 views

How define a reward function for a humanoid agent whose goal is to stand up from the ground?

I'm trying to teach a humanoid agent how to stand up after falling. The episode starts with the agent lying on the floor with its back touching the ground, and its goal is to stand up in the shortest ...
4
votes
0answers
75 views

Why is there inconsistency in the definitions of the retrace?

In Section 4.3 of paper Learning by Playing - Solving Sparse Reward Tasks from Scratch, the authors define Retrace as $$ Q^{ret}=\sum_{j=i}^\infty\left(\gamma^{j-i}\prod_{k=i}^jc_k\right)[r(s_j,a_j)+\...
4
votes
0answers
45 views

What research has been done on learning non-Markovian reward functions?

Recently, some work has been done planning and learning in Non-Markovian Decision Processes, that is, decision-making with temporally extended rewards. In these settings, a particular reward is ...
4
votes
1answer
138 views

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

Twin Delayed Deep Deterministic (TD3) policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained ...
4
votes
0answers
127 views

What is a generalized MDP?

What is a generalized MDP? Here are a few sub-questions. How is it different than a "regular" MDP? How does it generalize the notion of an MDP? Why do we need a generalized MDP? Do ...
4
votes
0answers
81 views

Is there any theoretical capabilities of apply deep successor representations with A3C algorithm?

Deep Successor Representations(DSR) has given better performance in tasks like navigation when it compares to normal model-free RL tasks. Basically, DSR is a hybrid of model-free RL and Model-Based RL....
4
votes
1answer
2k views

How to deal with different actions for different states of the environment?

I'm new to this AI/Machine Learning and was playing around with OpenAI Gym a bit. When looking through the environments, I came across the Blackjack-v0 environment, ...
4
votes
1answer
771 views

Traveling salesman problem variant: which algorithm to choose?

I have an industrial problem which I'm trying to cast as a Traveling Salesman problem (TSP) in 3D euclidian space. There are physical limitations which implies that some subpaths may or may not be ...
3
votes
0answers
41 views

How do neural networks deal with inputs of different sizes that are padded in order to have them of the same size?

I am trying to create an environment for RL where the size of my input (observation space) is not fixed. As a way around it, I thought about padding the size to a maximum value and then assigning &...
3
votes
1answer
84 views

How does sharing parameters between the policy and value functions help in PPO?

The PPO objective may include a value function error term when parameters are shared between the policy and value functions. How does this help, and when to use a neural network architecture that ...
3
votes
0answers
107 views

How to deal with a moving target in the Lunar Lander environment with DDPG?

I have noticed that DDPG does rather well at solving environments with a static target. For example, the default of Lunar Lander, the flags do not change position. So the DDPG model learns how to get ...
3
votes
0answers
62 views

Why do we discount the state distribution?

In Reinforcement Learning, it is common to use a discount factor $\gamma$ to give less importance to future rewards when calculating the returns. I have also seen mention of discounted state ...
3
votes
0answers
81 views

How do I implement the cross-entropy-method for a RL environment with a continuous action space?

I found many tutorials and posts on how to solve RL environments with discrete action spaces using the cross entropy method (e.g., in this blog post for the OpenAI Gym frozen lake environment). ...
3
votes
0answers
48 views

Difference between a distribution model and a sampling environment in Reinforcement Learning

The book from Sutton and Barto define a model in Reinforcement Learning as "something that mimics the behavior of the environment, or more generally, that allows inferences to be made about how ...
3
votes
1answer
69 views

How can I compress the states of a reinforcement learning agent?

I'm working on a problem that involves an RL agent with very large states. These states consist of several pieces of information about the agent. The states are not images, so techniques like ...
3
votes
0answers
57 views

Reinforcement learning and Graph Neural Networks: Entropy drops to zero

I am currently working on an experiment to link reinforcement learning with graph neural networks. This is my architecture: Feature Extraction with GCN: there is a fully meshed topology with ...
3
votes
0answers
74 views

Where does this variation of the importance sampling weight come from?

I have seeing a variation in importance sampling (IS) in Prioritized Experience Replay (PER) in some implementations regarding the original paper approach stated as (in section 3.4): $$ w_{i}=\left(\...
3
votes
0answers
61 views

How to implement REINFORCE with eligibility traces?

The pseudocode below is taken from Barto and Sutton's "Reinforcement Learning: an introduction". It shows an actor-critic implementation with eligibility traces. My question is: if I set $\...
3
votes
0answers
70 views

$\nabla \log \pi$ with respect to some parameters constantly being zero

I am new to reinforcement learning. May I ask a simple (and maybe a bit silly) question here? I am trying to use the "one-step actor-critic" method to train a robot on a gridworld. Let's ...
3
votes
0answers
23 views

Understanding neural network achitectures in policy gradient reinforcement learning for continuous state and action space

I am trying to train a neural network using reinforcement learning / policy gradient methods. The states, i.e. the inputs, as well as the actions I am trying to sample are vectors with each element ...
3
votes
0answers
32 views

What is the appropriate way of passing a list of integers that represents the environment to a neural network's dense layer?

I'm training an RL agent using the DQN algorithm to do a specific task. The environment is represented by a list of $10$ integer numbers from $0$ to $20$. An example would be $[5, 15, 8, 8, 0, \dots]$....
3
votes
0answers
174 views

In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?

While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6) $$ J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\...
3
votes
0answers
54 views

Why is it necessary to divide the priority range according to the batch size in Prioritized Experience Replay?

According to DeepMinds's paper Prioritized Experience Replay (2016), specifically Appendix B.2.1 "Proportional prioritization" (p. 13), one should equally divide the priority range $[0, p_\...
3
votes
0answers
39 views

Why weighting by lambda that sums to 1 ensures convergence in eligibility trace?

In Sutton and Barto's Book in chapter 12, they state that if weights sum to 1, then an equation's updates have "guaranteed convergence properties". Actually why it ensures convergence? There ...
3
votes
0answers
21 views

Mapping given probabilities to empirical probabilities

Consider following problem statement: You have given $n$ actions. You can perform any of them. Each action gives you success with some probability. The challenge is to perform given finite number of ...
3
votes
0answers
34 views

Why is it the case that off-policy evaluation using importance sampling suffers from high variance?

The average return for trajectories, $V^{\pi_e}$(s) is often computed via the importance sampling estimate $$V^{\pi_e}(s) = \frac{1}{n}\sum_{i=1}^n\prod_{t=0}^{H}\frac{\pi_e(a_t | s_t)}{\pi_b(a_t|s_t)}...
3
votes
0answers
84 views

How to take actions at each episode and within each step of the episode in deep Q learning?

In deep Q learning, we execute the algorithm for each episode, and for each step within an episode, we take an action and record a reward. I have a situation where my action is 2-tuple $a=(a_1,a_2)$. ...
3
votes
0answers
67 views

Convergence of a delayed policy update Q-learning

I thought about an algorithm that twists the standard Q-learning slightly, but I am not sure whether convergence to the optimal Q-value could be guaranteed. The algorithm starts with an initial ...
3
votes
0answers
34 views

How do reinforcement learning and collaborative learning overlap?

How do reinforcement learning and collaborative learning overlap? What are the differences and similarities between these fields? I feel like the results I get via google do not make the distinction ...
3
votes
1answer
93 views

Do smaller loss values during DQN training produce better policies?

During the training of DQN, I noticed that the model with prioritized experience replay (PER) had a smaller loss in general compared to a DQN without PER. The mean squared loss was an order of ...
3
votes
0answers
63 views

Simplification of expected reward under the limit in continuous tasks

I was reading the average reward setting for continuous tasks from rich sutton's book (page 202, 2nd edition). There he perform a simplification over the expected reward under the limit approaching to ...
3
votes
0answers
44 views

Why do we update $W$ with $\frac{1}{\mu (A_t | S_t)}$ instead of $\frac{\pi (A_t | S_t)}{\mu (A_t | S_t)}$ in off-policy Monte Carlo control?

I had the same question when I am reading the RL textbook from Sutton Bartol as posted here. Why do we update $W$ with $\frac{1}{\mu (A_t | S_t)}$ instead of $\frac{\pi (A_t | S_t)}{\mu (A_t | S_t)}$?...
3
votes
0answers
81 views

Understanding the TensorFlow implementation of the policy gradient method

I was trying to understand the implementation of a basic policy gradient (REINFORCE) method using TensorFlow. I think I got almost everything. The only thing that still bothers me is the loss function ...

1
2 3 4 5
14