# Unclear step in off-policy (every-visit) MC Control: why do we need the line: $A_t \neq \pi(S_t)$ then exit inner loop?

Could please some expert on reinforcement learning explain the red-box part in the following off-policy MC control? I mean I did not understand what (and why) is done in the step shown as a red-box. I understood all points, except the red box part: why do we need the line "$$A_t \neq \pi(S_t)$$ then exit inner loop"? More specifically, why don't we not exit the inner loop when $$A_t= \pi(S_t)$$?

• I suppose to more explicitly get at what is probably the primary point of confusion you could add something like: "Why can we not continue using the remainder of experience generated within this episode for lower time steps $t$?" (assuming that that is indeed what you're thinking) Commented Feb 2, 2022 at 20:19

why do we need the line "𝐴𝑡!=𝜋(𝑆𝑡) then exit inner loop"?

To understand this, you need to go back to the importance sampling ratio:

$$\rho_{t:t+n} = \prod_{k=1}^n \frac{\pi(A_{t+k}|S_{t+k})}{b(A_{t+k}|S_{t+k})}$$

NB Here I am using the more general stochastic form for the target policy $$\pi(a|s): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \mathbf{Pr}\{A_t = a | S_t = s \}$$. Whilst the pseudocode uses $$\pi(s): \mathcal{S} \rightarrow \mathcal{A}$$. You have to convert between them to see how the weighting factor is really working.

The weighting factor $$W$$ in the pseudocode is in fact $$\rho_{t:T-1}$$, calculated step by step working backwards through the trajectory so that it can be used to weight the return from timestep $$t$$ for each $$(s_t, a_t)$$ pair.

An important detail is that if any of the $$\pi(A_{t+k}|S_{t+k}) = 0$$ within an episode, then $$\rho_{t:T-1} = 0$$ also, and this must then be true for all other timesteps before $$t$$.

The line:

$$W \leftarrow W \frac{1}{b(A_t|S_t)}$$

assumes that $$\pi(A_{t}|S_{t}) = 1$$. This is true for the greedy policy if, and only if $$A_t$$ is the current greedy choice. As the code uses notation for deterministic policy, then that is the same as saying $$\pi(S_{t}) = A_t$$ from the pseudocode.

So, when $$A_t \neq \pi(S_t)$$, $$W$$ should be set to $$0$$, and this will logically make $$W = 0$$ throughout the rest of the loop, and there will be no updates to any Q values.

Exiting the loop when $$A_t \neq \pi(S_t)$$ saves you from adjusting $$W = 0$$ in the next line conditionally, then following through with calculating all the useless $$0$$ updates for earlier time steps.

Or another way to put it: With the $$W$$ update of $$W \leftarrow W \frac{1}{b(A_t|S_t)}$$, the loop may continue backwards to estimate expected returns from earlier timesteps only if the agent chose a non-exploring action compared to the current target policy. If an exploring action was taken, then the $$W$$ update from the pseudocode would be wrong. Not only would it be wrong, but determining the correct value - $$W = 0$$ - would result in no changes to any estimates for the rest of the loop. Easier just to terminate the loop at that point.

This is why off-policy Monte Carlo control is said to "only learn from the tails of episodes". If the agent explored a new action, that is learned from, but only from the point after it took the last exploring action to the end of the episode. Earlier exploring actions are also learned from indirectly, as they can put the agent in a different state that may not be reachable by the current target policy, and may turn out to be interesting.

• @ Neil Slater, the point I was asking was this: why do we need to check $A_t \neq \pi(S_t)$? Commented Feb 2, 2022 at 17:53
• @ Neil Slater, I changed the question photo with pseudocode from SB's book. Could you please update your answer based on this new photo? I think the updated answer will be okay for me. Commented Feb 2, 2022 at 17:59
• @ Neil Slatter. Thank you so much. You are TOTALLY an expert on RL! Commented Feb 3, 2022 at 18:18