Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
27
votes
5answers
11k views

What's the difference between model-free and model-based reinforcement learning?

What's the difference between model-free and model-based reinforcement learning? It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In ...
19
votes
4answers
5k views

How to handle invalid moves in reinforcement learning?

I want to create an AI which can play five-in-a-row/gomoku. As I mentioned in the title, I want to use reinforcement learning for this. I use policy gradient method, namely REINFORCE, with baseline. ...
19
votes
1answer
4k views

What is the relation between Q-learning and policy gradients methods?

As far as I understand, Q-learning and policy gradients (PG) are the two major approaches used to solve RL problems. While Q-learning aims to predict the reward of a certain action taken in a certain ...
13
votes
2answers
4k views

What is sample efficiency, and how can importance sampling be used to achieve it?

For instance, the title of this paper reads: "Sample Efficient Actor-Critic with Experience Replay". What is sample efficiency, and how can importance sampling be used to achieve it?
13
votes
2answers
4k views

How to define states in reinforcement learning?

I am studying reinforcement learning and the variants of it. I am starting to get an understanding of how the algorithms work and how they apply to an MDP. What I don't understand is the process of ...
12
votes
1answer
371 views

When should I use Reinforcement Learning vs PID Control?

When designing solutions to problems such as the Lunar Lander on OpenAIGym, Reinforcement Learning is a tempting means of giving the agent adequate action control so as to successfully land. But ...
12
votes
3answers
6k views

Are there any applications of reinforcement learning other than games?

Is there a way to teach reinforcement learning in applications other than games? The only examples I can find on the Internet are of game agents. I understand that VNC's control the input to the ...
11
votes
1answer
1k views

Why does DQN require two different networks?

I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...
11
votes
3answers
2k views

How to implement a constrained action space in reinforcement learning?

I'm coding a reinforcement learning model with a PPO agent thanks to the very good Tensorforce library, built on top of Tensorflow. The first version was very simple and I'm now diving into a more ...
10
votes
3answers
729 views

Why does the discount rate in the REINFORCE algorithm appear twice?

I was reading the book Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto (complete draft, November 5, 2017). On page 271, the pseudo-code for the episodic Monte-Carlo ...
10
votes
4answers
344 views

Can a neural network work out the concept of distance?

Imagine a game where it is a black screen apart from a red pixel and a blue pixel. Given this game to a human, they will first see that pressing the arrow keys will move the red pixel. The next thing ...
9
votes
2answers
232 views

Why doesn't Q-learning converge when using function approximation?

The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions regarding the learning rate are satisfied $\sum_{t} \alpha_t(s, a) = \infty$ $...
9
votes
1answer
215 views

How to stay a up-to-date researcher in ML/RL community?

As a student who wants to work on machine learning, I would like to know how it is possible to start my studies and how to follow it to stay up-to-date. For example, I am willing to work on RL and MAB ...
9
votes
4answers
933 views

Is the optimal policy always stochastic if the environment is also stochastic?

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? Intuitively, if the environment is ...
9
votes
2answers
555 views

Why is baseline conditional on state at some timestep unbiased?

In robotics, the reinforcement learning technique is used for finding the control pattern for a robot. Unfortunately, most policy gradient method are statistically biased which could bring the robot ...
9
votes
1answer
1k views

Why do you not see dropout layers on reinforcement learning examples?

I've been looking at reinforcement learning, and specifically playing around with creating my own environments to use with the OpenAI Gym AI. I am using agents from the stable_baselines project to ...
9
votes
1answer
1k views

A few doubts regarding the application of reinforcement learning to games like chess

I invented a chess-like board game. I built an engine so that it can play autonomously. The engine is basically a decision tree. It's composed by: A search function that at each node finds all ...
9
votes
1answer
5k views

Policy gradients for multiple continuous actions

Question is regarding Deep Reinforcement Learning using Policy Gradients. Cutting edge policy gradients algorithms are TRPO (Trusted Region Policy Optimization) and PPO (Proximal Policy Optimization)....
8
votes
5answers
324 views

What's a good resource for getting familiar with reinforcement learning?

I am familiar with supervised and unsupervised learning. I did the SaaS course done by Andrew Ng on Coursera.org. I am looking for something similar for reinforcement learning. Can you recommend ...
8
votes
2answers
421 views

Does Monte Carlo Search (specifically used by AlphaZero) Qualify as Machine Learning?

To the best of my understanding, Monte Carlo Search is an alternative method to Minimax for searching a tree of nodes. It works by choosing a move (generally the one with the highest chance of being ...
8
votes
1answer
1k views

What is the Bellman operator in reinforcement learning?

In mathematics, the word operator can refer to several distinct but related concepts. An operator can be defined as a function between two vector spaces, it can be defined as function where the domain ...
8
votes
1answer
210 views

Are there any other machine learning models apart from Reinforcement Learning and Q Learning to play video games?

OpenAI's Universe utilises RL algorithms and I have heard of some game-training projects using Q learning, but are there any others which are used to master/win games? Can genetic algorithms be used ...
8
votes
1answer
258 views

Getting to understand continuous state/action spaces MDPs and Reinforcement Learning

Most introductions to the field of MDPs and Reinforcement learning focus exclusively on domains where space and action variables are integers (and finite). This way we are introduced quickly to Value ...
7
votes
3answers
10k views

What are different actions in action space of environment of 'Pong-v0' game from openai gym?

Printing actionspace for Pong-v0 gives 'Discrete(6)' as output, i.e.0,1,2,3,4,5 are actions defined in environment as per documentation, but game needs only two controls. Why this discrepency? Further ...
7
votes
3answers
1k views

Why does is make sense to normalize rewards per episode in reinforcement learning?

In Open AI's actor-critic and in Open AI's REINFORCE, the rewards are being normalized like so rewards = (rewards - rewards.mean()) / (rewards.std() + eps) on ...
7
votes
2answers
4k views

What is the difference between Actor-Critic and Advantage Actor-Critic?

I'm struggling to understand the difference between Actor-Critic and Advantage Actor-Critic. At least I know they are different from Asynchronous Advantage Actor-Critic (A3C), as A3C adds ...
7
votes
1answer
5k views

How does LSTM in deep reinforcement learning differ from experience replay?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the author processed the Atari game frames with an LSTM layer at the end. My questions are: How does this method differ from the ...
7
votes
2answers
2k views

Inconsistent action space in Reinforcement Learning

This question is regarding Reinforcement Learning and different/inconsistent action space for every/some states. What do I mean by inconsistent action space? Let say you have an MDP where the number ...
7
votes
1answer
110 views

Are there reinforcement learning algorithms that scale to large problems?

Given a large problem, value iteration and other table based approaches seem to require too many iterations before they start to converge. Are there other reinforcement learning approaches that better ...
7
votes
2answers
426 views

What does “stationary” mean in the context of reinforcement learning?

I think I've seen the expressions "stationary data", "stationary dynamics" and "stationary policy", among others, in the context of reinforcement learning. What does it mean? I think stationary policy ...
7
votes
1answer
1k views

Do off-policy policy gradient methods exist?

Do off-policy policy gradient methods exist? I know that policy gradient methods themselves using the policy function for sampling rollouts. But can't we easily have a model for sampling from the ...
7
votes
1answer
280 views

Is there a difference in the architecture of deep reinforcement learning when multiple actions are performed instead of a single action?

I've built a deep deterministic policy gradient reinforcement learning agent to be able to handle any games / tasks that have only one action. However, the agent seems to fail horribly when there are ...
6
votes
3answers
778 views

Board/Card Game AI - Questions concerning state/action space - Deep Reinforcement Learning

Ok, I now know how a machine can learn to play to play Atari games (Breakout): Playing Atari with Reinforcement Learning With the same technique it is even possible to play FPS games (Doom): Playing ...
6
votes
1answer
3k views

What is the difference between an observation and a state in reinforcement learning?

I'm studying reinforcement learning. It seems that "state" and "observation" mean exactly the same thing. They both capture the current state of the game. Is there a difference between the two terms?...
6
votes
1answer
719 views

Learning Rate Decay and Exploration Rate Decay

Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
6
votes
1answer
179 views

Is Experience Replay like dreaming?

Drawing parallels between Machine Learning techniques and a human brain is a dangerous operation. When it is done successfully, it can be a powerful tool for vulgarisation, but when it is done with no ...
6
votes
1answer
153 views

Does AlphaZero use Q-Learning?

I was reading the AlphaZero paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, and it seems they don't mention Q-Learning anywhere. So does AZ use Q-...
6
votes
1answer
340 views

How do we prove the n-step return error reduction property?

In section 7.1 (about the n-step bootstrapping) of the book Reinforcement Learning: An Introduction (2nd edition), by Andrew Barto and Richard S. Sutton, the authors write about what they call the "n-...
6
votes
2answers
342 views

What is experience replay in laymen's terms?

I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...
6
votes
2answers
406 views

How should I handle action selection in the terminal state when implementing SARSA?

I recently started learning about reinforcement learning and currently I am trying to implement the SARSA algorithm, however I do not know how to deal with $Q(s', a')$, when $s'$ is the terminal state....
6
votes
1answer
93 views

Q-Learning the generic maze solution

After doing some exercices on Q-learning for maze solving, I wondered : my q-learning algorithms solve only ONE maze. The AI doesn't learn how to solve mazes, so how can I achieve it ? For instance ...
6
votes
1answer
1k views

OpenAI Baselines DQN - handling of invalid actions

I created an OpenAI Gym environment, and I would like to check the performance of the agent from OpenAI Baselines DQN approach on it. In my environment, the best possible outcome for the agent is 0 -...
6
votes
1answer
73 views

Purpose of Actor in Actor-Critic algorithm?

For discrete action spaces, what is the purpose of the actor in Actor-Critic algorithms? My current understanding is that the critic estimates the future reward given an action, so why not just take ...
6
votes
1answer
346 views

2 Player Games in OpenAI Retro

I have been using OpenAI Retro for awhile, and I wanted to experiment with two player games. By two player games, I mean co-op games like "Tennis-Atari2600" or even Pong, where 2 agents are present in ...
6
votes
2answers
534 views

Is it possible to implement reinforcement learning using a neural network?

I've implemented the reinforcement learning algorithm for an agent to play snappy bird (a shameless cheap ripoff of flappy bird) utilizing a q-table for storing the history for future lookups. It ...
6
votes
2answers
384 views

Reinforcement Learning in Commercial Strategy Games

I'm a professional game developer investigating the potential for using reinforcement learning to build strategy game AI opponents that have more creative behavior compared to traditional techniques ...
6
votes
1answer
180 views

Issue with simple game AI

A few months ago I made a simple game that is similar to the dinosaur game in Google Chrome - you jump over obstacles, or don't jump over levitating obstacles, and jump to collect bitcoins, which can ...
6
votes
1answer
162 views

Why does Bellman Equation solve an indirect policy?

I was watching a lecture on policy gradients vs Bellman equations. And they say that the Bellman equation indirectly creates a policy. While the policy gradient directly learns a policy? Why is this?
6
votes
1answer
201 views

A solution for a famous problem in RL

I'm here to ask you for a solution on this problem which is: how to use Reinforcement Learning in Immersive Virtual Reality to make a person move to a specific location in a virtual environment. As ...
5
votes
2answers
958 views

What algorithms are considered reinforcement learning algorithms?

What are the areas that belong to the Reinforcement Learning? TD(0), Q-Learning and SARSA are all temporal-difference algorithms, which belong to the reinforcement learning area, but is there more to ...