Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
3
votes
1answer
119 views

At what point are AWS GPU instances worth it compared to CPU, price-wise? [closed]

Let's say for: Image tasks Deep RL in high dimensional state space
1
vote
0answers
62 views

Impact of Varying Length Trajectories on Policy Gradient Optimization

As the question states, I am wondering how, if at all, a varying length of a trajectory (series of state,action pairs) will impact training/performance of policy gradient algorithms such as PPO, TRPO ...
1
vote
0answers
51 views

What limitations does the Markov property place on real time learning?

The Markov property is the dependence of a system's future state probability distribution solely on the present state, excluding any dependence on past system history. The presence of the Markov ...
2
votes
1answer
388 views

Using the opponent's mixed strategy in estimating the state value in minimax Q learning

In the paper Markov games as a framework for multi-agent reinforcement learning (which introduces the minimax Q Learning algorithm), at the bottom left of page 3, my understanding is that the author ...
5
votes
2answers
282 views

How can the importance sampling ratio be different than zero when the target policy is deterministic?

In the book Reinforcement Learning: An Introduction (2nd edition) Sutton and Barto define at page 104 (p. 126 of the pdf), equation (5.3), the importance sampling ratio, $\rho _{t:T-1}$, as follows: $$...
3
votes
2answers
899 views

Is Monte Carlo Tree Search appropriate for problems with large state and action spaces?

I'm doing a research on a finite-horizon Markov decision process with $t=1, \dots, 40$ periods. In every time step $t$, the (only) agent has to chose an action $a(t) \in A(t)$, while the agent is in ...
2
votes
1answer
92 views

What is the meaning of Model(s, a) in the prioritized sweeping algorithm?

I'm reading the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton). The authors provide the pseudocode of the prioritized sweeping algorithm, but I do not know ...
2
votes
1answer
162 views

Why is the actor-critic algorithm limited to using on-policy data?

Why is the actor-critic algorithm limited to using on-policy data? Or can we use the actor-critic algorithm with off-policy data?
1
vote
1answer
56 views

How do I calculate $max_{a′}Q(s′,a′,w−)$ when it is represented as a neural network?

Consider the following loss function $$ L(\mathbf{w}) = [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})]^2 $$ where $Q(s, a, \mathbf{w^-})$ and $Q(s, a, \mathbf{w})$ are ...
4
votes
3answers
560 views

How to train a logical XOR with reinforcement learning?

After reading an excellent BLOG post Deep Reinforcement Learning: Pong from Pixels and playing with the code a little, I've tried to do something simple: use the same code to train a logical XOR gate. ...
2
votes
1answer
226 views

Concrete Example for Q Learning

I am not sure if I understood the q learning algorithms correctly. Therefore I would give a concrete example and ask if someone can tell me how to update the q value correctly. First I initialized ...
0
votes
1answer
372 views

How do I update the Q values of a Deep Q Network when exploring?

I am trying to implement a Deep Q Network to play Asteroids. Unfortunately, I am not sure how to calculate the Q value exactly, if I am exploring. For example, the agent is exploring for 1 second (...
5
votes
3answers
1k views

What is a high dimensional state in reinforcement learning?

In the DQN paper, it is written that the state-space is high dimensional. I am a little bit confused about this terminology. Suppose my state is a high dimensional vector of length $N$, where $N$ is a ...
4
votes
2answers
1k views

How to implement a Continuous Control of a quadruped robot with Deep Reinforcement Learning in Pybullet and OpenAI Gym?

Description I have designed this robot in URDF format and its environment in pybullet. Each leg has a minimum and maximum value of movement. What reinforcement algorithm will be best to create a ...
3
votes
2answers
357 views

How can I design a reinforcement learning model for a game with multiple complex actions taken at a time?

I have a steady hex-map and turn-based wargame featuring WWII carrier battles. On a given turn, a player may choose to perform a large number of actions. Actions can be of many different types, and ...
2
votes
1answer
304 views

Is Python and a single computer sufficient to implement and train an RL-based agent for a turn-based war game? [closed]

I have a steady hex-map and turn-based wargame featuring WWII carrier battles. I would like to improve the given AI part of the game using reinforcement learning. I have a bunch of noob questions. Is ...
5
votes
1answer
2k views

Why does the "reward to go" trick in policy gradient methods work?

In the policy gradient method, there's a trick to reduce the variance of policy gradient. We use causality, and remove part of the sum over rewards so that only actions happened after the reward are ...
5
votes
1answer
1k views

What is the difference between imitation learning and classification done by experts?

In short, imitation learning means learning from the experts. Suppose I have a dataset with labels based on the actions of experts. I use a simple binary classifier algorithm to assess whether it is ...
1
vote
2answers
618 views

SARSA won't work for linear function approximator for MountainCar-v0 in OpenAI environment. What are the possible causes?

I am learning Reinforcement Learning from the lectures from David Silver. I finished lecture 6 and went on to try SARSA with linear function approximator for MountainCar-v0 environment from OpenAI. A ...
2
votes
1answer
414 views

How should I define the reward function in the case of Connect Four?

I'm using RL to train a Network on the game Connect4. It learns quickly that 4 connected pieces is good. It gets a reward of 1 for this. A zero is rewarded for all other moves. It takes quite a time ...
1
vote
2answers
232 views

In the context of importance sampling ratio, how is the equation $\mathbb{E}\left[\rho_{t: T-1} G_{t} | S_{t}=s\right]=v_{\pi}(s)$ derived?

When reading the book by Sutton and Barto, I came across the importance sampling ratio. The first equation, I believe, describes the probability a particular sequence is obtained given the current ...
2
votes
1answer
583 views

Why is the reward signal normalized in openAI's REINFORCE? [duplicate]

Pytorch's example for the REINFORCE algorithm for reinforcement learning has the following code: ...
2
votes
1answer
142 views

How does reinforcement learning handle measured disturbances?

I recently encountered an interesting problem and was wondering how RL would solve it. The objective of the problem is to maximize the coffee quality, given by box X. The coffee quality objective ...
6
votes
1answer
1k views

Why is the n-step tree backup algorithm an off-policy algorithm?

In reinforcement learning book from Sutton & Barto (2018 edition), specifically in section 7.5 of the book, they present an n-step off-policy algorithm that doesn't require importance sampling ...
2
votes
1answer
126 views

Reinforcement Learning with Adaptive Action Magnitude

How to select an action in a state if the action does not necesarily cause the environment to change state? Given 10 states ($S_0$ to $S_9$) and in each state $i$ there are two actions defined $(1,-...
17
votes
2answers
7k views

Any papers regarding different/inconsistent action space in Reinforcement Learning? [closed]

This question is regarding Reinforcement Learning and different/inconsistent action space for every/some states. What do I mean by different/inconsistent action space? Let say you have an MDP where ...
2
votes
0answers
165 views

Reward discounting in reinforcement learning for a Pong game

I am trying to understand how to train a neural network to win a Pong game using reinforcement learning, by following the blog post Spinning up a Pong AI with deep reinforcement learning. The ...
3
votes
2answers
605 views

What is the main difference between additive rewards and discounted rewards?

What is the difference between additive and discounted rewards?
11
votes
2answers
1k views

How do we prove the n-step return error reduction property?

In section 7.1 (about the n-step bootstrapping) of the book Reinforcement Learning: An Introduction (2nd edition), by Andrew Barto and Richard S. Sutton, the authors write about what they call the "n-...
1
vote
1answer
591 views

DQN exploration strategy for large grid-world environment

My task involves a large grid-world type of environment (grid size may be $30\times30$, $50\times50$, $100\times100$, at the largest $200\times200$). Each element in this grid either contains a 0 or a ...
1
vote
1answer
272 views

How do we actually sample an action from a policy in policy gradient methods?

Recently I started to look at policy gradient methods and policies are represented as functions with features for larger problems with many states. Many articles and pseudocodes of algorithms mention ...
2
votes
1answer
1k views

Neural Network Optimizers in Reinforcement Learning non-well behaved environments

https://stackoverflow.com/questions/36162180/gradient-descent-vs-adagrad-vs-momentum-in-tensorflow Here, the nice gifs explain how different algorithms approach towards the root. Unfortunately, the ...
1
vote
2answers
269 views

For forecasting and trading control, given limited data, what AI approaches are well matched?

I'm working on stock price prediction and automatic or semi-automatic control of trading. The price trends of these stocks exhibit recurring patterns that may be exploited. My dataset is currently ...
2
votes
1answer
151 views

Understanding lemma 2 of the "Trust Region Policy Optimization" paper

In the Trust Region Policy Optimization paper, in Lemma 2 of Appendix A, I did not quite understand deriving inequality (31) from (30), which is: $$\bar{A}(s) = P(a \neq \tilde{a} | s) \mathbb{E}_{(a,...
5
votes
3answers
972 views

How to deal with episode termination in Advantage Actor-Critic algorithm?

Advantage Actor-Critic algorithm may use the following expression to get 1-step estimate of the advantage: $ A(s_t,a_t) = r(s_t, a_t) + \gamma V(s_{t+1}) (1 - done_{t+1}) - V(s_t) $ where $done_{t+...
0
votes
1answer
955 views

DQN it's not working properly

I'm trying to build a DQN to replicate the DeepMind results. I'm doing with a simple DQN for the moment, but it isn't learning properly: after +5000 episodes, it couldn't get more than 9-10 points. ...
3
votes
1answer
1k views

Scrabble game using machine learning

I've been thinking if machine learning can be used to play the game Scrabble. My knowledge is limited in the ML field, thus I've seeking some pointers :) I want to know how could I possibly build a ...
1
vote
1answer
110 views

Why is it ok to calculate the reward based on a hidden state?

I'm looking at this source code, where the reward is calculated with reward = cmp(score(self.player), score(self.dealer)) Why is it ok to calculate the reward ...
3
votes
0answers
117 views

Why are all the actions converging to the same index?

I am using PPO with an LSTM agent. My agent is performing 10 actions for each episode, one action is corresponding to one LSTM timestep and the action space is discrete. I have only one reward per ...
2
votes
1answer
401 views

Why does Q-learning converge to the optimal policy, even if the agent acts sub-optimally?

In Q-learning, during training, it doesn't matter how the agent selects actions. The algorithm always converges to the optimal policy. Why does this happen? What's the intuition?
0
votes
1answer
390 views

Is it possible to use a feed-forward neural network to predict the actions in reinforcement learning?

I have done a lot of research on the internet about Reinforcement Learning and I found encountered methods of Reinforcement Learning: Q-Learning and Deep Q-Learning. And I have developed a vague idea ...
2
votes
1answer
1k views

Why are Q values updated according to the greedy policy?

Apparently, in the Q-learning algorithm, the Q values are not updated according to the "current policy", but according to a "greedy policy". Why is that the case? I think this is related to the fact ...
2
votes
1answer
4k views

What is the time complexity of the value iteration algorithm?

Recently, I have come across the information (lecture 8 and 9 about MDPs of this UC Berkeley AI course) that the time complexity for each iteration of the value iteration algorithm is $\mathcal{O}(|S|^...
3
votes
1answer
850 views

Deep Q-Learning poor convergence on Stochastic Environment

I'm trying to implement a Deep Q-network in Keras/TF that learns to play Minesweeper (our stochastic environment). I have noticed that the agent learns to play the game pretty well with both small and ...
3
votes
1answer
118 views

What is the best way to integrate unchangeable ethics into a chatbot

I am building a generative model chatbot as a research and learning project. One of the most important parts of my project is to research ways in which I can make this chatbot work in a consistently ...
2
votes
1answer
131 views

Do we need the transition probability function when calculating the importance sampling ratio?

I am reading the book titled "Reinforcement Learning: An Introduction" (by Sutton and Barto). I am at chapter 5, which is about Monte Carlo methods, but now I am quite confused. There is one thing I ...
2
votes
1answer
63 views

Reinforcement Learning to Grouped Scheduling Optimisation Problem

I am not sure the name of this kind of problem, but anyway, the situation is as below. Assign teachers into Groups and consider on each of their workload, availability etc. There are some other soft/...
3
votes
1answer
212 views

How does Hindsight Experience Replay learn from unsuccessful trajectories?

I am confused by how HER learns from unsuccessful trajectories. I understand that from failed trajectories it creates 'fake' goals that it can learn from. Ignoring HER for now, if in the case where ...
1
vote
1answer
50 views

Are artificial intelligence learnings or trainings transferable from one agent to the other?

One disadvantage or weakness of Artificial Intelligence today the slow nature of learning or training success. For instance, an AI agent might require a 100,000 samples or more to reach an appreciable ...
4
votes
1answer
307 views

Building AI from chess - data shape from simulation [closed]

Problem My problem is the following: Given 1000 wins, losses, and ties from a chess simulation I am using, what shape should each game be (I.e., sequence of moves leading to win/loss/tie) in order to ...

1
30 31
32
33 34
37