Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
5
votes
1answer
169 views

How do I recognise a bandit problem?

I'm having difficulty understanding the distinction between a bandit problem and a non-bandit problem. An example of the bandit problem is an agent playing $n$ slot machines with the goal of ...
2
votes
2answers
82 views

How to convert sequences of images into state in DQN?

I recently read the DQN paper titled: Playing Atari with Deep Reinforcement Learning. My basic and rough understanding of the paper is as follows: You have two neural networks; one stays frozen for a ...
14
votes
3answers
843 views

Why doesn't Q-learning converge when using function approximation?

The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions (the Robbins-Monro conditions) regarding the learning rate are satisfied $\...
2
votes
0answers
29 views

If the minimum Q value is decreasing and the maximum Q value increasing, is this a sign that dueling double DQN is diverging?

I'm training a dueling double DQN agent with prioritized replay buffer and notice that the min Q values are decreasing, while the max Q values are increasing. Is this a sign that it is diverging? ...
3
votes
1answer
49 views

Simplification of expected reward under the limit in continuous tasks

I was reading the average reward setting for continuous tasks from rich sutton's book (page 202, 2nd edition). There he perform a simplification over the expected reward under the limit approaching to ...
4
votes
1answer
48 views

How can a single sample represent the expectation in gradient temporal difference learning?

I was reading the gradient temporal difference learning version 2(GTD2) from rich Sutton's book page-246. At some point, he expressed the whole expectation using a single sample from the environment. ...
2
votes
1answer
63 views

How does $\mathbb{E}$ suddenly change to $\mathbb{E}_{\pi'}$ in this equation?

In Sutton-Barto's book on page 63 (81 of the pdf): $$\mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_t=s,A_t=\pi'(s)] = \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_{t} = s]$$ How does $...
3
votes
1answer
62 views

What is the solution to the exercise 3.11 in the RL book by Sutton and Barto

I'm trying to solve the exercise 3.11 from the second-edition from the Sutton and Barto's book Exercise 3.11 If the current state is $S_t$ , and actions are selected according to a stochastic ...
2
votes
1answer
56 views

Why does AlphaGo Zero select move based on exponentiated visit count?

From the AlphaGo Zero paper, AlphaGo Zero uses an exponentiated visit count from the tree search. Why use visit count instead of the mean action value $Q(s, a)$?
1
vote
1answer
81 views

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

I have a problem which I believe can be described as a contextual bandit. More specifically, in each round, I observe a context from the environment consisting of five continuous features, and, ...
8
votes
1answer
295 views

What are some resources on continuous state and action spaces MDPs for reinforcement learning?

Most introductions to the field of MDPs and Reinforcement learning focus exclusively on domains where space and action variables are integers (and finite). This way we are introduced quickly to Value ...
2
votes
1answer
120 views

How to train a reinforcement learning agent from raw pixels?

How would you train a reinforcement learning agent from raw pixels? For example, if you have 3 stacked images to sense motion, then how would you pass them to neural networks to output Q-learning ...
3
votes
0answers
68 views

How to take actions at each episode and within each step of the episode in deep Q learning?

In deep Q learning, we execute the algorithm for each episode, and for each step within an episode, we take an action and record a reward. I have a situation where my action is 2-tuple $a=(a_1,a_2)$. ...
3
votes
1answer
38 views

What's the right way of building a deep Q-network?

I'm new to RL and to deep q-learning and I have a simple question about the architecture of the neural network to use in an environment with a continous state space a discrete action space. I tought ...
3
votes
2answers
55 views

Is there any difference between reward and return in reinforcement learning?

I am reading Sutton and Barto's book on reinforcement learning. I thought that reward and return were the same things. However, in Section 5.6 of the book, 3rd line, first paragraph, it is written: ...
4
votes
2answers
88 views

Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

In equation 3.17 of Sutton and Barto's book: $$q_*(s, a)=\mathbb{E}[R_{t+1} + \gamma v_*(S_{t+1}) \mid S_t = s, A_t = a]$$ $G_{t+1}$ here have been replaced with $v_*(S_{t+1})$, but no reason has ...
1
vote
1answer
41 views

Why do we also need to normalize the action's values on continuous action spaces?

I was reading here tips & tricks for training in DRL and I noticed the following: always normalize your observation space when you can, i.e., when you know the boundaries normalize your ...
3
votes
4answers
367 views

Reinforcement Learning (RL) how to obtain $p(s',r|s,a)$

I am trying to study the book Reinforcement Learning: An Introduction (Sutton & Barto, 2018). In chapter 3.1 the authors state the following exercise Exercise 3.5 Give a table analogous to that ...
0
votes
0answers
25 views

How to choose hyperparameters in double DQN?

I'm looking for some indications about the tuning of hyperparameters in building my double DQN. I have a time series problem (with about 2000 observations and no terminal state, I have to max the ...
1
vote
2answers
54 views

Are there any good tutorials about training RL agent from raw pixels using PyTorch?

Is there any good tutorials about training reinforcement learning agent from raw pixels using PyTorch? I don't understand the official PyTorch tutorial. I want to train the agent on the atari ...
2
votes
1answer
49 views

What should the action space for the card game Crib be?

I'm working on creating an environment for a card game, which the agent chooses to discard certain cards in the first phase of the game, and uses the remaining cards to play with. (The game is Crib if ...
4
votes
1answer
81 views

Why not more TD(𝜆) in actor-critic algorithms?

Is there either an empirical or theoretical reason that actor-critic algorithms with eligibility traces have not been more fully explored? I was hoping to find a paper or implementation or both for ...
1
vote
0answers
18 views

Is the TD-residual defined for timesteps $t$ past the length of the episode?

Let $\mathcal{S}$ be the state-space in a reinforcement learning problem where rewards are in $\mathbb{R}$, and let $V:\mathcal{S} \to \mathbb{R}$ be an approximate value function. Following the GAE ...
5
votes
2answers
470 views

Why are lambda returns so rarely used in policy gradients?

I've seen monte-carlo reward $G_{t}$ used in REINFORCE and TD($0$) reward $r_t + \gamma Q(s', a')$ used in vanilla actor-critic. I've never seen someone use lambda reward $G^{\lambda}_{t}$ in these ...
0
votes
0answers
29 views

How is the general return-based off-policy equation derived?

I'm wondering how is the general return-based off-policy equation in Safe and efficient off-policy reinforcement learning derived $$\mathcal{R} Q(x, a):=Q(x, a)+\mathbb{E}_{\mu}\left[\sum_{t \geq 0} \...
5
votes
2answers
417 views

Why am I getting the incorrect value of lambda?

I am trying to solve for $\lambda$ using temporal-difference learning. More specifically, I am trying to figure out what $\lambda$ I need, such that $\text{TD}(\lambda)=\text{TD}(1)$, after one ...
1
vote
0answers
96 views

When do you back-propagate errors through a neural network when using TD($\lambda$)?

I have a neural network that I'm want to use to self-play Connect Four. The neural network receives the board state and is to provide an estimate of the state's value. I would then, for each move, ...
4
votes
1answer
1k views

Can TD($\lambda$) be used with deep reinforcement learning?

TD lambda is a way to interpolate between TD(0) - bootstrapping over a single step, and, TD(max), bootstrapping over the entire episode length, or, Monte Carlo. Reading the link above, I see that an ...
5
votes
1answer
119 views

What is the intuition behind TD($\lambda$)?

I'd like to better understand temporal-difference learning. In particular, I'm wondering if it is prudent to think about TD($\lambda$) as a type of "truncated" Monte Carlo learning?
1
vote
0answers
62 views

How can the target rely on untrained parameters?

I'm trying to understand DQN. I understand where the loss function comes from. I'm just unsure about why the target function works in practice. Given the loss function $$ L_i(\theta_i) = [(y_i - Q(s,a;...
2
votes
1answer
55 views

How is $\Delta$ updated in true online TD($\lambda$)?

In the RL textbook by Sutton & Barto section 7.4, the author talked about the "True online TD($\lambda$)". The figure (7.10 in the book) below shows the algorithm. At the end of each step, $V_{...
0
votes
1answer
38 views

What is correct update when the some indexes are not available?

To update the Q table Q-learning takes the arg max of the Q values - the state, value mappings. For example, in tic tac toe the state XOX OXO -X- contains two ...
1
vote
1answer
48 views

Shouldn't the utility function of two-player zero-sum games be in the range $[-1, 1]$?

In Appendix B of MuZero, they say In two-player zero-sum games the value functions are assumed to be bounded within the $[0, 1]$ interval. I'm confused about the boundary: Shouldn't the value/...
3
votes
1answer
41 views

What happens if the opponent doesn't play optimally in minimax?

I just read an article about the minimax algorithm. When you design the algorithm, you assume that your opponent is a perfect player, i.e. it plays optimally. Let's consider the game of chess. What ...
8
votes
1answer
169 views

Are Q-learning and SARSA the same when action selection is greedy?

I'm currently studying reinforcement learning and I'm having difficulties with question 6.12 in Sutton and Barto's book. Suppose action selection is greedy. Is Q-learning then exactly the same ...
9
votes
3answers
636 views

Does Monte Carlo tree search qualify as machine learning?

To the best of my understanding, the Monte Carlo tree search (MCTS) algorithm is an alternative to minimax for searching a tree of nodes. It works by choosing a move (generally, the one with the ...
0
votes
0answers
33 views

How can I combine word2vec with tf-idf to have concatenated features?

I want to develop a focused crawler using deep reinforcement learning and a priority queue that will work as the crawler frontier. I reckon using x = (state, action)...
0
votes
0answers
15 views

Is there any advantage to using a non-diagonal covariance matrix for a policy distribution?

For reinforcement learning implementations with a gym.spaces.Box action space, which is the product of $k$ real closed intervals, it is common (actually more like ...
1
vote
1answer
74 views

Can tabular Q-learning converge even if it doesn't explore all state-action pairs?

My understanding of tabular Q-learning is that it essentially builds a dictionary of state-action pairs, so as to maximize the Markovian (i.e., step-wise, history-agnostic?) reward. This incremental ...
2
votes
1answer
50 views

What is the difference between reinforcement learning and evolutionary algorithms?

What is the difference between reinforcement learning (RL) and evolutionary algorithms (EA)? For some problems, you could presumably co-evolve two "species" populations using evolutionary algorithms ...
1
vote
0answers
15 views

How to learn how to select a subgraph via reinforcement learning?

I have the following problem. I am given a graph with a lot (>30000) nodes. Nodes are associated with a low (<10)-dimensional feature vector, and edges are associated with a low (<10)-...
2
votes
0answers
24 views

Are the final states not being updated in this $n$-step Q-Learning algorithm?

I am reading this paper and in algorithm 3 they describe an $n$-step Q-Learning algorithm. Below is the pseudo-code. From this pseudo-code, it looks as though the final tuples that they would ...
1
vote
1answer
55 views

Is there any programming practice website for beginners in Reinforcement Learning [closed]

I am doing an online course on Reinforcement Learning from university of Alberta. It focus too much on theory. I am engineering and I am interested towards applying RL to my applications directly. ...
1
vote
0answers
39 views

Calculating the advantage 'gain' of actions in model-free reinforcement learning

I have a simple question about model-free reinforcement. In a model I'm writing about, I want to know the value 'gain' we'd get for executing an action, relative to the current state. That is, what ...
2
votes
1answer
32 views

Can the agent wait until the end of the episode to determine the reward in SARSA?

From Sutton and Barto's book Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 99), the following definition for first-visit MC prediction, for estimating $V \sim V_\pi$ is ...
1
vote
1answer
53 views

Is the distribution of state-action pairs from sample based planning accurate for small experience sets?

From the David Silver's lecture 8: Integrating Learning and Planning - based on Sutton and Barto - he talks about using sample-based planning to use our model to take a sample of a state and then use ...
2
votes
1answer
30 views

Adversarial Q Learning should use the same Q Table?

I'm creating a RF Q-Learning agent for a two player fully-observable board game and wondered, if I was to train the Q Table using adversarial training, should I let both 'players' use, and update, the ...
1
vote
1answer
78 views

What is a RAM state in the gym's breakout-ram environment?

I have encountered the gym environment and decided to create AI that plays breakout. Here is the link: https://gym.openai.com/envs/Breakout-ram-v0/. The documentation says that the state is ...
1
vote
0answers
29 views

Actor-Critic implementation not learning

I've implemented a vanilla actor-critic and have run into a wall. My model does not seem to be learning the optimal policy. The red graph below shows its performance in cartpole, where the algorithm ...
1
vote
1answer
94 views

Handle non-existing states in q-learning

I am using Q-learning to solve an engineering problem. The objective is to generate a Q-table associating state to Q-values. I created a State vector ...

1 2 3
4
5
25