Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

468 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2
votes
0answers
278 views

Implementation of PPO - Value Loss not converging, return plateauing

Copy from my reddit post: (Sorry if this does not fit here, please tell me and i delete it) Help regarding I'm working on an implementation of PPO, which i plan to use in my (Bachelors) Thesis. To ...
2
votes
0answers
107 views

Hashed Tile Coding vs Regular Tile Coding

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto explain at page 221 a form of tile coding using hashing, to reduce memory consumption. I have two questions about that: ...
2
votes
0answers
28 views

Are there reinforcement learning algorithms that ensure convergence for continuous state space problems?

The Q-learning does not guarantee convergence for continuous state space problems (Why doesn't Q-learning converge when using function approximation?). In that case, is there an algorithm which ...
2
votes
0answers
15 views

How is GARB implemented in PGRD-DL to calculate gradients w.r.t. internal rewards?

In section 3 of this paper the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator. I have read it a number of times and ...
2
votes
0answers
40 views

How does the TRPO surrogate loss account for the error in the policy?

In the Trust Region Policy Optimization (TRPO) paper, on page 10, it is stated An informal overview is as follows. Our proof relies on the notion of coupling, where we jointly define the ...
2
votes
0answers
194 views

Why overfitting is bad in DQN?

It is mentioned by Fu 2019 that overfitting might have a negative effect on training DQN. They showed that with either early stopping or experience replay this effect could be reduced. The first is ...
2
votes
0answers
76 views

Why experience reply memory in DQN instead of a RNN memory?

I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
2
votes
0answers
222 views

Difficulty understanding Monte Carlo policy evaluation (state-value) for gridworld

I've been trying to read Sutton & Barto book chapter 5.1, but I'm still a bit confused about the procedure of using Monte Carlo policy evaluation (p.92), and now I just cant proceed anymore coding ...
2
votes
0answers
36 views

Hindsight Experience Replay with multiple goals

What if there are multiple goals? For example, let's consider Bit-flipping environment as described in the paper HER with one small change: Now, goal is not some specific configuration, but let's say ...
2
votes
0answers
77 views

Reinforcement learning for inventory management with dynamic changes to available products

Consider a shop owner who has to deal with having to buy for one week from a different supplier with several different brands. Another week a brand is removed or added from the market. Yet another ...
2
votes
0answers
150 views

Is Reinforcement Learning the future of Natural Language Processing?

I was reading about the grounding problem after seeing it mentioned in another answer today. The article states that, in order to avoid the "infinite regress" of defining all words with other words, ...
2
votes
0answers
113 views

How to choose method for solving planning problems?

There are many methods and algorithms dealing with planning problems. If I understand correctly, according to Wikipedia, there are classical planning problems, with: a unique known initial state, ...
2
votes
0answers
59 views

Eligibility trace In Model-based Reinforcement Learning

In model-based reinforcement learning algorithms, the model of the environment is constructed to efficiently use samples, models such as Dyna, and Prioritize Sweeping. Moreover, eligibility trace ...
2
votes
0answers
34 views

How to include exploration in Gaussian policy

When dealing with continuous action spaces, a common choice when designing a policy in policy gradient methods is to learn mean and variance of actions for a specific state and then simply sample from ...
2
votes
0answers
24 views

At what point are AWS GPU instances worth it compared to CPU, *price wise*?

Let's say for: 1. Image tasks 2. Deep RL in high dimensional state space
2
votes
1answer
192 views

Large and Multiple-actions space

I have a steady hex-map and turn-based wargame featuring WWII carrier battles On a given turn, a player may choose different and independent actions 
(moving one, two naval unit, assigning a mission ...
2
votes
0answers
48 views

Understanding the proof of theorem 2.1 from the paper “Efficient reductions for imitation learning”

I am trying to understand the proof of theorem 2.1 from this paper: Ross, Stéphane, and Drew Bagnell. "Efficient reductions for imitation learning." Proceedings of the thirteenth international ...
2
votes
0answers
278 views

Solving equations using reinforcement learning

I was lately curious about a reinforcement learning approach that would solve maths equations. For example, if I have the following equation: $$ f(g(h(w))) = 0 , with \ w = \begin{matrix} a_{11} &...
2
votes
0answers
316 views

How many episodes does it take for a vanilla one-step actor-critic agent to master the OpenAI BipedalWalker-v2 problem?

I'm trying to solve the OpenAI BipedalWalker-v2 by using a one-step actor-critic agent. I'm implementing the solution using python and tensorflow. I'm following this pseudo-code taken from the book ...
2
votes
0answers
66 views

Why do we have to solve MDP in each iteration of Maximum Entropy Inverse Reinforcement Learning?

Gradient in Maximum Entropy IRL requires to find the probability of expert trajectories given the reward function weights. This is done in the paper by calculating state visitation probabilities but I ...
2
votes
0answers
69 views

Algorithms for multiple agents problems

Can anyone recommend a reinforcement learning algorithm for a multi-agent environment? In my simplified example, I'm implementing a Q-Learning system with different 10 agents. The agents compete for ...
2
votes
0answers
76 views

When can we say an RL algorithm learns an Atari game?

If an Atari game's rewards can be between $-100$ and $100$, when can we say an agent learned to play this game? Should it get the reward very close to $100$ for each instance of the game? Or it is ...
2
votes
0answers
240 views

Extend the loss function from the single action to the n-action case per time step

My question concerns a side question (which was not answered) asked here: Policy gradients for multiple continuous actions I am trying to implement a simple policy gradient algorithm for a discrete ...
2
votes
0answers
86 views

Which features and algorithm could optimize this air-conditioner problem?

Imagine we have 2 air conditioner systems (AA) and 2 "free cooling" systems which mix external and internal air (FC) in a closed box which always tends to warm up. For each system, we have to find ...
2
votes
0answers
170 views

Is iLQG a good algorithm for model-based planning with simple environments?

In their work Continuous Deep Q-Learning with Model-based Acceleration, the author demonstrate great results of applying Imagination Rollouts for model-based acceleration of learning. They test their ...
2
votes
1answer
203 views

Is it expected that during self-play reinforcement learning that player 1 or player 2 wins the majority of games?

I'm testing various learning rates and neural network configurations. I'm testing over 10000 games, with the first 2000 having random starting moves and general randomness throughout of about 20%, i.e....
1
vote
0answers
17 views

Trying to proof off policy TD Learning formula

I was reading the book "Introduction to Reinforcement Learning" by Richard Sutton In section 7.3 he write the formula for n-step off-policy TD as:. $$V(S_t) = V(S_{t-1}) + \alpha \rho_{t:t+n-...
1
vote
0answers
27 views

Should I use the discounted average reward as objective in a finite-horizon problem?

I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
1
vote
0answers
20 views

When past states contain useful information, does A3C perform better than TD3, given that TD3 does not use an LSTM?

I am trying to build an AI that needs to have some information about the past states as well. Therefore, LSTMs are suitable for this. Now, I want to know that for a problem/game like Breakout, where ...
1
vote
0answers
32 views

How can deep Q-learning converge if the targets may not be correct?

In deep Q-learning, $Q(s, a)$ and $Q'(s, a)$ are predicted or estimated by the neural network itself. In supervised learning, the target value is a true unbiased value. However, this isn't the case in ...
1
vote
0answers
31 views

What does self-play in reinforcement learning lead to?

Suppose, instead of playing against a random opponent, the reinforcement learning algorithm described above played against itself, with both sides learning. What do you think would happen in this case?...
1
vote
0answers
67 views

What happens if our target network overestimates the value?

When we use DDQN, we often use the target network in case our online network overestimates a value, but this doesn't make sense to me, because What happens if our target network is the one that ...
1
vote
0answers
49 views

What is the best way to make a deep reinforcement learning environment with a continuous 2D action space?

I understand that the actor-critic method is probably where I want to start because of how it works with continuous action spaces. However, the problem I am trying to solve would require the action be ...
1
vote
0answers
60 views

Most of state-action pairs remain unvisited in the q-table

In building my first Q-learning algorithm for OpenAI gym's CartPole problem, many of my states remain unvisited. I believe it is the reason that my agent does not learn. Can I be told of the reasons I ...
1
vote
0answers
37 views

What is the difference between Bayes-adaptive MDP and a Belief-MDP in Reinforcement Learning?

I have been reading a few papers in this area recently and I keep coming across these two terms. As far as I'm aware, Belief-MDPs are when you cast a POMDP as a regular MDP with a continuous state ...
1
vote
0answers
51 views

When to apply reward for time series data?

Reading the paper 'Reinforcement Learning for FX trading 'at https://stanford.edu/class/msande448/2019/Final_reports/gr2.pdf it states: While our end goal is to be able to make decisions on a ...
1
vote
0answers
39 views

What is a multi channel supervised classifier?

I came across a paper that describes its model architecture in the following way. Our TRIL network is a two-channel network jointly trained to predict the expert’s action given state and the system’s ...
1
vote
0answers
44 views

Classification or regression for deep Q learning

DQN implemented at https://github.com/PacktPublishing/PyTorch-1.x-Reinforcement-Learning-Cookbook/blob/master/Chapter07/chapter7/dqn.py uses the mean square error loss function for the neural network ...
1
vote
0answers
51 views

In continuous action spaces, how is the standard deviation, associated with Gaussian distribution from which actions are sampled, represented?

I have a question about implementing policy gradient methods for problems with continuous action spaces. Assume that actions are sampled from a diagonal Gaussian distribution with mean vector $\mu$ ...
1
vote
0answers
35 views

How can I convert a simple CLI RPG to a compatible environment for training an RL agent via stable-baselines?

What would be the good choice of algorithm to use for character action selection in an RPG, implemented in Python? I had previously asked this question in the hopes of getting headway on the AI ...
1
vote
0answers
44 views

Correct dimensionality of parameter vector for solving an MRP with linear function approximation?

I'm in the process of trying to learn more about RL by shadowing a course offered collaboratively by UCL and DeepMind that has been made available to the public. I'm most of the way through the course,...
1
vote
0answers
27 views

Solving multi-armed bandit problems with continuous action space

My problem has a single state and an infinite amount of actions on a certain interval (0,1). After quite some time of googling I found a few paper about an algorithm called zooming algorithm which can ...
1
vote
0answers
23 views

Should the importance sampling ratio be updated at the end of the for loop in the off-policy Monte Carlo control algorithm?

I'm studying RL with Sutton and Barto's book. I'd like to ask about the order of execution of a statement in the algorithm below. Here, $W$ (importance sampling ratio) is updated at the end of the <...
1
vote
0answers
31 views

What is the role of embeddings in a deep recurrent Q network?

When describing the model architecture for a deep recurrent q network, the authors of the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning each agent consists of a recurrent ...
1
vote
0answers
36 views

Do we need multiple parallel environments to train in batches an on-policy algorithm?

When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ...
1
vote
0answers
22 views

Tic-tac-toe: How would standard SARSA and Q-learning yield different results in the agent's behaviour?

I know this is deceptively simple. Tic tac toe is a well studied game for RL. Assume your agent is playing aggainst a strong opponent. I know you deal in after states. I know that in Q learning the ...
1
vote
0answers
29 views

How are the coefficients of the Region of Interest being selected?

I was reading the following paper: Rl-Ncs: Reinforcement Learning Based Data-Driven Approach For Nonuniform Compressed Sensing, and my question is: how do they decide whether a signal is characterized ...
1
vote
0answers
45 views

Atari Games: Pretrained CNN to accelerate training?

DQN for Atari takes considerable training time. For example, the 2015 paper in Nature notes that algorithms are trained for 50 million frames or equivalently around 38 days of game experience in total....
1
vote
0answers
32 views

Off-Policy Estimation - Importance Sampling with Negative Rewards

Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and ...
1
vote
0answers
26 views

What is the proof that the variance of the gradient estimate in Actor-Critic is smaller than in REINFORCE?

The intuition provided when introducing actor-critic algorithms is that the variance of its gradient estimates is smaller than in REINFORCE as, e.g., discussed here. This intuition makes sense for the ...

1 2 3
4
5
10