4 votes
Accepted

Unclear paragraph in Sutton-Barto on exploration/exploitation relating to bandit like decision tasks

The quoted text is from the end of a paragraph that is explaining some aspects of bandit algorithms, and looking forward to how knowledge of them is applicable in a reinforcement learning context. ...
Neil Slater's user avatar
  • 31.7k
4 votes

Would AlphaZero perform better if made with transformers?

Yes! LeelaChessZero, an open-source re-implementation and continuation of AlphaZero, has been experimenting with this for a while now. Their strongest networks are currently transformers, not ...
Todd Sewell's user avatar
3 votes
Accepted

Unclear point in derivation of action-value function

As $q_{\pi}(s,a)$ is defined in S&B RL book as follows: we define the value of taking action a in state s under a policy $\pi$, denoted $q_{\pi}(s,a)$, as the expected return starting from $s$, ...
cinch's user avatar
  • 1,323
3 votes
Accepted

REINFORCE with Baseline update rule

The value $\delta$ is already representing a derivative equivalent to derivative of MSE loss for the difference between observed and predicted return. Multiplying it by the gradient of $\hat{v}$ to ...
Neil Slater's user avatar
  • 31.7k
3 votes
Accepted

Should I use RL for this allocation problem and, if yes, which RL approach?

I was just reading Sutton and Barto last night and a passage stuck with me which I will paraphrase. RL is marked by the evaluation of an action, not the instruction of an action. So I would say if ...
foreverska's user avatar
2 votes
Accepted

How can reward at time step $t$ can be a function of a state at time step $t+1$?

There is a free choice when deciding the indexing of rewards in reinforcement learning. You can decide to associate the immediate reward with the time step of state and action that generated it, or ...
Neil Slater's user avatar
  • 31.7k
2 votes

Expectile regression in Implicit Q-Learning

Because $argmin_{m_\tau} E_{x\sim X} \left[L_2^\tau(x - m_\tau)\right]$ approximates $max_{x \in X} \hspace{0.1cm} x$, then $E_{(s,a,s')\sim \mathcal{D}}\left[\left(max_{a'\in \mathcal{A}\;s.t. \pi_\...
user118967's user avatar
2 votes
Accepted

How are POMDPs solved in practice?

A formal POMDP approach that models a belief state (a distribution over possible states) needs to have a theoretical model for the hidden aspects of the state. This is not always appropriate or ...
Neil Slater's user avatar
  • 31.7k
2 votes
Accepted

Is there any example of a Markov Decision Process (MDP) with infinite number of states?

Many real-world problems have an infinite number of states, but, in practice, digital computers cannot represent an infinite number of states or numbers anyway, so you will always need to discretize ...
nbro's user avatar
  • 40.2k
2 votes
Accepted

How does changing the order of a sequence of rewards affect the cumulative discounted reward

The relation is actually $$ G_{\text{asc}} \leq G \leq G_{\text{dec}} $$ Without loss of generality, we can say that the sequence $r_0, r_1, .., r_n = G_{\text{dec}}$ is already sorted in descending ...
vl_knd's user avatar
  • 474
2 votes
Accepted

Why should the agent bounce the ball back and forth on the same side of the screen in Atari Breakout?

There might be some cases, say, once the agent clearing a brick obtaining a positive reward and then forever bouncing the ball on the same side of the screen would have a higher cumulative reward than ...
cinch's user avatar
  • 1,323
1 vote
Accepted

Do I really need to do policy evaluation until convergence in policy iteration?

I don't understand why policy evaluation needs to be done until convergence It doesn't need to be, although the resulting algorithm if you cut short of convergence is not strictly policy iteration as ...
Neil Slater's user avatar
  • 31.7k
1 vote

If $p(s'|s,a) = 0$, would the reward the reward $r(s,a,s')$ be infinite?

You've correctly figured out the most part of your confusion. The only thing I want to add is that by the finite MDP definition here it's certainly possible $p(s'|s,a)=0$ for some $s',s$. The reason ...
cinch's user avatar
  • 1,323
1 vote

If $p(s'|s,a) = 0$, would the reward the reward $r(s,a,s')$ be infinite?

Ok, so I was able to find the answer to this question by myself. So I'm sharing it with everyone. By the law of non-exclusive events, $P(A|B) = \frac{P(A,B)}{P(B)}$ where $P(B) > 0$. This can also ...
Jahid Chowdhury Choton's user avatar
1 vote
Accepted

Gymnasium/Petting Zoo: Creating a copy of the board/env

So how am I supposed to consider hypothetical board positions starting from a given state? In an environment designed for model-free reinforcement learning, you cannot, at least not via any of the ...
Neil Slater's user avatar
  • 31.7k
1 vote

What does the notation $\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$ appearing in Generalized Advantage Estimation mean?

In policy gradient methods reducing bias is crucial to obtain more accurate gradient estimates for updating the policy in a stochastic gradient ascent fashion. The usual advantage function as you ...
cinch's user avatar
  • 1,323
1 vote
Accepted

What is the logic in including/not including subscript $\pi$ in in "E" for value functions?

Your second equation is defined on page 78 and in the same page there's a step on the lower part of derivations contains the answer to your confusion. $q_{\pi}(s, {\pi'}(s)) \\= E[R_{t+1} + \gamma v_{...
cinch's user avatar
  • 1,323
1 vote

What is the logic in including/not including subscript $\pi$ in in "E" for value functions?

The subscript just indicates over which variable the expectation is taken. For consistency it can be included everywhere. Sometimes, when the variable over which the expectation is taken is obvious, ...
vl_knd's user avatar
  • 474
1 vote
Accepted

Is it possible to describe wordle as a multi armed bandit problem?

Is it possible to describe wordle as a multi armed bandit problem? Yes, although the data management to present the current choice would involve constructing a state much like a reinforcement ...
Neil Slater's user avatar
  • 31.7k
1 vote

Batch wise Inference to speed up Muzero's MCTS

Batching: A Good Idea You're right, batching is a great way to speed up AlphaZero or MuZero self-play! Your proposed solution of running multiple games in parallel is the easiest way to achieve some ...
Todd Sewell's user avatar
1 vote

Where can I find good sources (textbook, lecture notes, etc) on multi-armed bandits?

There is the bandit book by Lattimore and Szepesvari, which is, I think, the main reference in the field: you can find pretty everything about Bandits, even more recent and research oriented topics. ...
Luca Anzalone's user avatar
1 vote

How to properly model the MDP of a weighted graph with the constraint of only visiting each vertex once (and not get stuck in infinite loops)?

Many environments are non-Markovian. Sometimes based on the perception of the agent a Markovian environment becomes non-Markovian (woods101 with perception aliasing). If the model assumes a Markovian ...
foreverska's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible