Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
69 votes
6 answers
71k views

What's the difference between model-free and model-based reinforcement learning?

What's the difference between model-free and model-based reinforcement learning? It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In ...
user avatar
39 votes
2 answers
18k views

What is the relation between Q-learning and policy gradients methods?

As far as I understand, Q-learning and policy gradients (PG) are the two major approaches used to solve RL problems. While Q-learning aims to predict the reward of a certain action taken in a certain ...
user avatar
  • 515
38 votes
5 answers
13k views

How should I handle invalid actions (when using REINFORCE)?

I want to create an AI which can play five-in-a-row/Gomoku. I want to use reinforcement learning for this. I use the policy gradient method, namely REINFORCE, with baseline. For the value and policy ...
user avatar
23 votes
1 answer
10k views

What is the Bellman operator in reinforcement learning?

In mathematics, the word operator can refer to several distinct but related concepts. An operator can be defined as a function between two vector spaces, it can be defined as a function where the ...
user avatar
  • 34.5k
22 votes
2 answers
13k views

What is sample efficiency, and how can importance sampling be used to achieve it?

For instance, the title of this paper reads: "Sample Efficient Actor-Critic with Experience Replay". What is sample efficiency, and how can importance sampling be used to achieve it?
user avatar
  • 373
22 votes
2 answers
9k views

Are there other approaches to deal with variable action spaces?

This question is about Reinforcement Learning and variable action spaces for every/some states. Variable action space Let's say you have an MDP, where the number of actions varies between states (for ...
user avatar
21 votes
3 answers
3k views

Why doesn't Q-learning converge when using function approximation?

The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions (the Robbins-Monro conditions) regarding the learning rate are satisfied $\...
user avatar
  • 34.5k
20 votes
1 answer
17k views

What is the credit assignment problem?

In reinforcement learning (RL), the credit assignment problem (CAP) seems to be an important problem. What is the CAP? Why is it relevant to RL?
user avatar
  • 34.5k
20 votes
1 answer
5k views

When should I use Reinforcement Learning vs PID Control?

When designing solutions to problems such as the Lunar Lander on OpenAIGym, Reinforcement Learning is a tempting means of giving the agent adequate action control so as to successfully land. But ...
user avatar
19 votes
2 answers
3k views

What is the "Hello World" problem of Reinforcement Learning?

As we all know, "Hello World" is usually the first program that any programmer learns/implements in any language/framework. As Aurélien Géron mentioned in his book that MNIST is often called ...
user avatar
18 votes
2 answers
14k views

How to define states in reinforcement learning?

I am studying reinforcement learning and the variants of it. I am starting to get an understanding of how the algorithms work and how they apply to an MDP. What I don't understand is the process of ...
user avatar
  • 283
17 votes
3 answers
8k views

What is the difference between actor-critic and advantage actor-critic?

I'm struggling to understand the difference between actor-critic and advantage actor-critic. At least, I know they are different from asynchronous advantage actor-critic (A3C), as A3C adds an ...
user avatar
  • 987
16 votes
2 answers
13k views

What is the difference between First-Visit Monte-Carlo and Every-Visit Monte-Carlo Policy Evaluation?

I came across these 2 algorithms, but I cannot understand the difference between these 2, both in terms of implementation as well as intuitionally. So, what difference does the second point in both ...
user avatar
16 votes
4 answers
2k views

Why does the discount rate in the REINFORCE algorithm appear twice?

I was reading the book Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto (complete draft, November 5, 2017). On page 271, the pseudo-code for the episodic Monte-Carlo ...
user avatar
16 votes
1 answer
16k views

How does LSTM in deep reinforcement learning differ from experience replay?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the author processed the Atari game frames with an LSTM layer at the end. My questions are: How does this method differ from the ...
user avatar
16 votes
1 answer
9k views

What is the difference between an observation and a state in reinforcement learning?

I'm studying reinforcement learning. It seems that "state" and "observation" mean exactly the same thing. They both capture the current state of the game. Is there a difference between the two terms?...
user avatar
  • 653
16 votes
2 answers
10k views

What is the difference between reinforcement learning and optimal control?

Coming from a process (optimal) control background, I have begun studying the field of deep reinforcement learning. Sutton & Barto (2015) state that particularly important (to the writing of the ...
user avatar
15 votes
1 answer
723 views

How to stay a up-to-date researcher in ML/RL community?

As a student who wants to work on machine learning, I would like to know how it is possible to start my studies and how to follow it to stay up-to-date. For example, I am willing to work on RL and MAB ...
user avatar
  • 451
15 votes
1 answer
5k views

Why do you not see dropout layers on reinforcement learning examples?

I've been looking at reinforcement learning, and specifically playing around with creating my own environments to use with the OpenAI Gym AI. I am using agents from the stable_baselines project to ...
user avatar
15 votes
1 answer
4k views

Why does DQN require two different networks?

I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...
user avatar
15 votes
3 answers
8k views

Are there any applications of reinforcement learning other than games?

Is there a way to teach reinforcement learning in applications other than games? The only examples I can find on the Internet are of game agents. I understand that VNC's control the input to the ...
user avatar
15 votes
3 answers
6k views

How to implement a variable action space in Proximal Policy Optimization?

I'm coding a Proximal Policy Optimization (PPO) agent with the Tensorforce library (which is built on top of TensorFlow). The first environment was very simple. Now, I'm diving into a more complex ...
user avatar
  • 153
15 votes
2 answers
10k views

Can Q-learning be used for continuous (state or action) spaces?

Many examples work with a table-based method for Q-learning. This may be suitable for a discrete state (observation) or action space, like a robot in a grid world, but is there a way to use Q-learning ...
user avatar
15 votes
1 answer
8k views

How can policy gradients be applied in the case of multiple continuous actions?

Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms. When using a single continuous action, normally, you would use some ...
user avatar
15 votes
1 answer
4k views

How to deal with a huge action space, where, at every step, there is a variable number of legal actions?

I am working on creating an RL-based AI for a certain board game. Just as a general overview of the game so that you understand what it's all about: It's a discrete turn-based game with a board of ...
user avatar
  • 325
14 votes
3 answers
2k views

Does Monte Carlo tree search qualify as machine learning?

To the best of my understanding, the Monte Carlo tree search (MCTS) algorithm is an alternative to minimax for searching a tree of nodes. It works by choosing a move (generally, the one with the ...
user avatar
14 votes
1 answer
2k views

Why does the policy network in AlphaZero work?

In AlphaZero, the policy network (or head of the network) maps game states to a distribution of the likelihood of taking each action. This distribution covers all possible actions from that state. ...
user avatar
14 votes
3 answers
6k views

What is a "trajectory" in reinforcement learning?

I'm now learning about reinforcement learning, but I just found the word "trajectory" in this answer. However, I'm not sure what it means. I read a few books on the Reinforcement Learning but none of ...
user avatar
  • 987
14 votes
2 answers
6k views

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

Q-learning uses a table to store all state-action pairs. Q-learning is a model-free RL algorithm, so how could there be the one called Deep Q-learning, as deep means using DNN; or maybe the state-...
user avatar
  • 1,213
14 votes
4 answers
5k views

What does "stationary" mean in the context of reinforcement learning?

I think I've seen the expressions "stationary data", "stationary dynamics" and "stationary policy", among others, in the context of reinforcement learning. What does it mean? I think stationary policy ...
user avatar
13 votes
3 answers
5k views

Why exactly do neural networks require i.i.d. data?

In reinforcement learning, successive states (actions and rewards) can be correlated. An experience replay buffer was used, in the DQN architecture, to avoid training the neural network (NN), which ...
user avatar
  • 34.5k
13 votes
3 answers
8k views

Why does is make sense to normalize rewards per episode in reinforcement learning?

In Open AI's actor-critic and in Open AI's REINFORCE, the rewards are being normalized like so rewards = (rewards - rewards.mean()) / (rewards.std() + eps) on ...
user avatar
  • 669
12 votes
3 answers
913 views

Why is the reward in reinforcement learning always a scalar?

I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
user avatar
  • 223
12 votes
1 answer
5k views

What is the relation between online (or offline) learning and on-policy (or off-policy) algorithms?

In the context of RL, there is the notion of on-policy and off-policy algorithms. I understand the difference between on-policy and off-policy algorithms. Moreover, in RL, there's also the notion of ...
user avatar
  • 34.5k
12 votes
3 answers
3k views

Is the optimal policy always stochastic if the environment is also stochastic?

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? Intuitively, if the environment is ...
user avatar
  • 34.5k
12 votes
4 answers
1k views

Counterexamples to the reward hypothesis

On Sutton and Barto's RL book, the reward hypothesis is stated as that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative ...
user avatar
  • 221
11 votes
2 answers
1k views

Why is reinforcement learning not the answer to AGI?

I previously asked a question about How can an AI freely make decisions?. I got a great answer about how current algorithms lack agency. The first thing I thought of was reinforcement learning, since ...
user avatar
  • 345
11 votes
2 answers
2k views

How do we prove the n-step return error reduction property?

In section 7.1 (about the n-step bootstrapping) of the book Reinforcement Learning: An Introduction (2nd edition), by Andrew Barto and Richard S. Sutton, the authors write about what they call the "n-...
user avatar
  • 111
11 votes
1 answer
1k views

Are Q-learning and SARSA the same when action selection is greedy?

I'm currently studying reinforcement learning and I'm having difficulties with question 6.12 in Sutton and Barto's book. Suppose action selection is greedy. Is Q-learning then exactly the same ...
user avatar
  • 131
11 votes
1 answer
2k views

How could I use reinforcement learning to solve a chess-like board game?

I invented a chess-like board game. I built an engine so that it can play autonomously. The engine is basically a decision tree. It's composed by: A search function that at each node finds all ...
user avatar
  • 255
10 votes
3 answers
1k views

What algorithms are considered reinforcement learning algorithms?

What are the areas/algorithms that belong to reinforcement learning? TD(0), Q-Learning and SARSA are all temporal-difference algorithms, which belong to the reinforcement learning area, but is there ...
user avatar
10 votes
4 answers
3k views

Are there any online competitions for Reinforcement Learning?

Kaggle is limited to only supervised learning problems. There used to be www.rl-competition.org but they've stopped. Is there anything else I can do other than locally trying out different algorithms ...
user avatar
10 votes
3 answers
17k views

What do the different actions of the OpenAI gym's environment of 'Pong-v0' represent? [closed]

Printing action_space for Pong-v0 gives Discrete(6) as output, i.e. $0, 1, 2, 3, 4, 5$ are actions defined in the environment as ...
user avatar
  • 211
10 votes
1 answer
3k views

What is the difference between expected return and value function?

I've seen numerous mathematical explanations of reward, value functions $V(s)$, and return functions. The reward provides an immediate return for being in a specific state. The better the reward, the ...
user avatar
10 votes
1 answer
2k views

Do off-policy policy gradient methods exist?

Do off-policy policy gradient methods exist? I know that policy gradient methods themselves using the policy function for sampling rollouts. But can't we easily have a model for sampling from the ...
user avatar
  • 653
10 votes
2 answers
557 views

Was DeepMind's DQN learning simultaneously all the Atari games?

DeepMind states that its deep Q-network (DQN) was able to continually adapt its behavior while learning to play 49 Atari games. After learning all games with the same neural net, was the agent able to ...
user avatar
  • 203
9 votes
5 answers
731 views

What's a good resource for getting familiar with reinforcement learning?

I am familiar with supervised and unsupervised learning. I did the SaaS course done by Andrew Ng on Coursera.org. I am looking for something similar for reinforcement learning. Can you recommend ...
user avatar
  • 213
9 votes
2 answers
1k views

What are the biggest barriers to get RL in production?

I am studying the state of the art of Reinforcement Learning, and my point is that we see so many applications in the real world using Supervised and Unsupervised learning algorithms in production, ...
user avatar
9 votes
1 answer
2k views

What is the difference between reinforcement learning and evolutionary algorithms?

What is the difference between reinforcement learning (RL) and evolutionary algorithms (EA)? I am trying to understand the basics of RL, but I do not yet have practical experience with RL. I know ...
user avatar
9 votes
2 answers
918 views

Why is baseline conditional on state at some timestep unbiased?

In the homework for the Berkeley RL class, problem 1, it asks you to show that the policy gradient is still unbiased if the baseline subtracted is a function of the state at time step $t$. $$ \...
user avatar
  • 91

1
2 3 4 5
42