4

A non-starving policy is a (behavior) policy that is theoretically guaranteed to visit each state and take all possible actions from each state an infinite number of times, so that to always update $Q(s, a)$, $\forall s, \forall a$, an infinite number of times. In the context of off-policy prediction, this criterion implies that any trajectory will have no ...


3

In respect of RL, is model-free and off-policy the same thing, just different terminology? No, they are entirely different terms, with the only thing they have in common is that they are both ways in which an RL agent can vary. An agent is generally either working off-policy or on-policy, and is generally either model-based or model-free. These things can ...


1

If your game agent performs any kind of advance learning from self play or database of moves, that will generate parameters for some kind of model (e.g. a table of expected values, or neural network weights to select a preferred action). This is unavoidable, and if you want to re-use the results of that machine learning, you absolutely have to store the ...


1

Multiplying the entire update by $\rho$ has the desirable property that experience affects $Q$ less when the behavior policy is unrelated to the target policy. In the extreme, if the trajectory taken has zero probability under the target policy, then $Q$ isn't updated at all, which is good. Alternatively, if only $G$ is scaled by $\rho$, taking zero ...


1

I don't think the section was written in haste. I think they just didn't have space to include the whole proof. It's a bit involved, so they just gave concepts. An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning gives a proof of stability. At least parts of it should seem familiar if you've read Sutton and Barto's proof of the ...


Only top voted, non community-wiki answers of a minimum length are eligible