The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
19
votes
4answers
5k views

How to handle invalid moves in reinforcement learning?

I want to create an AI which can play five-in-a-row/gomoku. As I mentioned in the title, I want to use reinforcement learning for this. I use policy gradient method, namely REINFORCE, with baseline. ...
-1
votes
1answer
119 views

What is the difference between a stochastic and a deterministic policy?

In reinforcement learning, there are the concepts of stochastic (or probabilistic) and deterministic policies. What is the difference between them?
7
votes
2answers
421 views

What does “stationary” mean in the context of reinforcement learning?

I think I've seen the expressions "stationary data", "stationary dynamics" and "stationary policy", among others, in the context of reinforcement learning. What does it mean? I think stationary policy ...
2
votes
1answer
267 views

Why are Q values updated according to the greedy policy?

Apparently, in the Q-learning algorithm, the Q values are not updated according to the "current policy", but according to a "greedy policy". Why is that the case? I think this is related to the fact ...
27
votes
5answers
11k views

What's the difference between model-free and model-based reinforcement learning?

What's the difference between model-free and model-based reinforcement learning? It seems to me that any model-free learner, learning through trial and error, could be reframed as model-based. In ...
13
votes
2answers
4k views

How to define states in reinforcement learning?

I am studying reinforcement learning and the variants of it. I am starting to get an understanding of how the algorithms work and how they apply to an MDP. What I don't understand is the process of ...
9
votes
1answer
5k views

Policy gradients for multiple continuous actions

Question is regarding Deep Reinforcement Learning using Policy Gradients. Cutting edge policy gradients algorithms are TRPO (Trusted Region Policy Optimization) and PPO (Proximal Policy Optimization)....
6
votes
1answer
5k views

How does LSTM in deep reinforcement learning differ from experience replay?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the author processed the Atari game frames with an LSTM layer at the end. My questions are: How does this method differ from the ...
11
votes
1answer
1k views

Why does DQN require two different networks?

I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...
3
votes
1answer
988 views

What is the difference between First-Visit Monte-Carlo and Every-Visit Monte-Carlo Policy Evaluation?

I came across this 2 algorithms but I cannot understand the difference between these 2 both in terms of implementation as well as intuitionally. So what difference does the second point in both the ...
6
votes
2answers
342 views

What is experience replay in laymen's terms?

I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...
4
votes
2answers
155 views

How can a reinforcement learning agent generalize if it is trained against only one opponent?

I started teaching myself about reinforcement learning a week ago and I have this confusion about the learning experience. Let's say we have the game Go. And we have an agent that we want to be able ...
2
votes
2answers
149 views

Concrete Example for Q Learning

I am not sure if I understood the q learning algorithms correctly. Therefore I would give a concrete example and ask if someone can tell me how to update the q value correctly. First I initialized ...
4
votes
2answers
127 views

Deciding on a reward per each action in a given state (Q-learning)

I looked for existing posts on Stack Exchange, which kind of answer the questions about the reward system and reward function, but not specifically what I want to ask here, which is how do you ...
4
votes
1answer
180 views

What is the credit assignment problem?

In reinforcement learning (RL), the credit assignment problem (CAP) seems to be an important problem. What is the CAP? Why is it relevant to RL?
2
votes
2answers
176 views

How to see terminal reward in self-play reinforcement learning?

there seems to be a major difference how the terminal reward is received/handled in self-play RL vs "normal" RL which confuses me. I implemented TicTacToe the normal way, where a single agent plays ...
8
votes
5answers
322 views

What's a good resource for getting familiar with reinforcement learning?

I am familiar with supervised and unsupervised learning. I did the SaaS course done by Andrew Ng on Coursera.org. I am looking for something similar for reinforcement learning. Can you recommend ...
8
votes
1answer
1k views

What is the Bellman operator in reinforcement learning?

In mathematics, the word operator can refer to several distinct but related concepts. An operator can be defined as a function between two vector spaces, it can be defined as function where the domain ...
13
votes
2answers
4k views

What is sample efficiency, and how can importance sampling be used to achieve it?

For instance, the title of this paper reads: "Sample Efficient Actor-Critic with Experience Replay". What is sample efficiency, and how can importance sampling be used to achieve it?
11
votes
3answers
2k views

How to implement a constrained action space in reinforcement learning?

I'm coding a reinforcement learning model with a PPO agent thanks to the very good Tensorforce library, built on top of Tensorflow. The first version was very simple and I'm now diving into a more ...
9
votes
4answers
926 views

Is the optimal policy always stochastic if the environment is also stochastic?

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? Intuitively, if the environment is ...
9
votes
2answers
229 views

Why doesn't Q-learning converge when using function approximation?

The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions regarding the learning rate are satisfied $\sum_{t} \alpha_t(s, a) = \infty$ $...
4
votes
1answer
315 views

Reinforcement Learning in asteroid game

Introduction An attractive asteroid game was described in the paper from 2007: quote: “In our first experiment, the virtual agent is a spaceship pilot, The pilot’s task is to maneuver the ...
6
votes
1answer
3k views

What is the difference between an observation and a state in reinforcement learning?

I'm studying reinforcement learning. It seems that "state" and "observation" mean exactly the same thing. They both capture the current state of the game. Is there a difference between the two terms?...
4
votes
1answer
594 views

Suitable reward function for trading buy and sell orders

I am working to build an deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are ...
7
votes
1answer
110 views

Are there reinforcement learning algorithms that scale to large problems?

Given a large problem, value iteration and other table based approaches seem to require too many iterations before they start to converge. Are there other reinforcement learning approaches that better ...
6
votes
1answer
339 views

How do we prove the n-step return error reduction property?

In section 7.1 (about the n-step bootstrapping) of the book Reinforcement Learning: An Introduction (2nd edition), by Andrew Barto and Richard S. Sutton, the authors write about what they call the "n-...
6
votes
3answers
772 views

Board/Card Game AI - Questions concerning state/action space - Deep Reinforcement Learning

Ok, I now know how a machine can learn to play to play Atari games (Breakout): Playing Atari with Reinforcement Learning With the same technique it is even possible to play FPS games (Doom): Playing ...
5
votes
1answer
274 views

Is the discount not needed in a deterministic environment for Reinforcement Learning?

I'm now reading a book titled as "Deep Reinforcement Learning Hands-On" and the author said the following on the chapter about AlphaGo Zero: Self-play In AlphaGo Zero, the NN is used to ...
4
votes
2answers
457 views

Why exactly do neural networks require i.i.d. data?

In reinforcement learning, in general, successive states (actions and rewards) are highly correlated. An "experience replay" buffer was used, in the DQN architecture, to avoid training the neural ...
3
votes
1answer
702 views

DQN input representation for a card game

In order to learn about DP and RL, I chose to start a side project where I would train an AI to play a "simple" card game. I will be doing this using the DQN with replay memory. The problem is, I can'...
3
votes
2answers
294 views

Each training run for DDQN agent takes 2 days, and still ends up with -13 avg score, but OpenAi baseline DQN needs only an hour to converge to +18?

Status: For a few weeks now, I have been working on a Double DQN agent for the PongDeterministic-v4 environment, which you can find here. A single training run ...
2
votes
2answers
1k views

Why does self-playing TicTacToe not become perfect?

I trained a DQN that learns TicTacToe by playing against itself with a reward of -1/0/+1 for a loss/draw/win. Every 500 episodes I test the progress by letting it play some episodes (also 500) against ...
2
votes
1answer
51 views

Can someone please help me validate my MDP?

Problem Statement : I have a system with four states - S1 through S4 where S1 is the beginning state and S4 is the end/terminal state. The next state is always better than the previous state i.e if ...
5
votes
1answer
79 views

What is the appropriate approach to playing a game with incomplete state information?

I have a steady hex-map and turn-based war game featuring WWII carrier battles. I would like to improve the fixed policy for the AI using reinforcement learning. I have some beginner's questions, ...
5
votes
2answers
675 views

What is the difference between reinforcement learning and optimal control?

Coming from a process (optimal) control background, I have begun studying the field of deep reinforcement learning. Sutton & Barto (2015) state that particularly important (to the writing of ...
4
votes
1answer
727 views

Traveling salesman problem variant: which algorithm to choose?

I have an industrial problem which I'm trying to cast as a Traveling Salesman problem (TSP) in 3D euclidian space. There are physical limitations which implies that some subpaths may or may not be ...
8
votes
2answers
415 views

Does Monte Carlo Search (specifically used by AlphaZero) Qualify as Machine Learning?

To the best of my understanding, Monte Carlo Search is an alternative method to Minimax for searching a tree of nodes. It works by choosing a move (generally the one with the highest chance of being ...
6
votes
2answers
404 views

How should I handle action selection in the terminal state when implementing SARSA?

I recently started learning about reinforcement learning and currently I am trying to implement the SARSA algorithm, however I do not know how to deal with $Q(s', a')$, when $s'$ is the terminal state....
3
votes
1answer
39 views

Expressing Arbitrary Reward Functions as Potential-Based Advice (PBA)

I am trying to reproduce the results for the simple grid-world environment in [1]. But it turns out that using a dynamically learned PBA makes the performance worse and I cannot obtain the results ...
3
votes
2answers
125 views

Is there any difference between a control and an action in reinforcement learning?

There are reinforcement learning papers (e.g. Metacontrol for Adaptive Imagination-Based Optimization) that use (apparently, interchangeably) the term control or action to refer to the effect of the ...
3
votes
0answers
95 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
2
votes
1answer
112 views

What information should be cached in experience replay for actor-critic?

Experience replay is buffer (or a "memory") of transactions $e_t = (s_t, a_t, r_t, s_{t+1})$. The equations for calculating the loss in actor critic are an actor loss (parameterized by $\theta$) $$...
1
vote
1answer
71 views

Deep Q Learning Algorithm for Simple Python Game makes player stuck

I made a simple Python game. A screenshot is below: Basically, a paddle moves left and right catching particles. Some make you lose points while others make you gains points. This is my first Deep Q ...
1
vote
1answer
73 views

In online one step actor critic, why does the weights update become less significant as the episode progresses?

The Reinforcement Learning Book by Richard Sutton et al, section 13.5 shows an online actor critic algorithm. Why do the weights updates depend on the discount factor via $I$? It seems that the more ...
1
vote
1answer
98 views

Importance of starting state and player in RL for Tic Tac Toe

So I am simulating a Tic Tac Toe game with a human opponent. The way the RL trains is through policy/value iterations for a fixed number of iterations all specified by the user. Now whether the human ...
1
vote
1answer
35 views

Can Fuzzy Logic Control Self-tune as Other Learning Systems Do?

This inquiry appeared in the comments to one of the answers of this question, but is actulaly not related to emotional intelligence, so it is reproduced here as a separate question I thought (maybe ...
1
vote
1answer
170 views

Why is the reward signal normalized in openAI's REINFORCE?

Pytorch's example for the REINFORCE algorithm for reinforcement learning has the following code: ...
0
votes
1answer
79 views

What will Q-values look like in self-play tic-tac-toe?

This corresponds to Exercise 1.1 of RLBook, and a discussion followed from here. Considering two reward schemes- Win = +1, Draw = 0, Loss = -1 Win = +1, Draw or Loss = 0 Can we say something about ...
0
votes
1answer
65 views

inconsistent formulas for loss calculation in OpenAI's Actor Critic?

Open Ai's (working) actor critic code calculates the losses like so: ...