Behaviour policy must be stochastic in states where it is not identical to the Optimal policy

Question

In Sutton & Barto Reinforcement learning book, page 103 (chapter: off-policy learning via importance sampling), the following statement is given:

"In order to use episodes from $b$ to estimate values for $\pi$, we require that every action taken under $\pi$ is also taken, at least occasionally, under $b$. That is, we require that $\pi$(a|s) > 0 implies $b$(a|s) > 0. This is called the assumption of coverage. It follows from coverage that $b$ must be stochastic in states where it is not identical to $\pi$. "

where $b$ refers to a behavior policy and $\pi$ to the target optimal policy. I don't understand how the above conclusion was obtained from the definition of coverage.

Answer 1

Say we are in state $s$, and $b$ chooses $a'$ but $\pi$ chooses $a$. Since $\pi$ chose $a$ ($\pi(a | s) >0$), coverage implies that $b(a|s)>0$. But $a'$ was ultimately chosen by $b$, so it must be that $b(a'|s)>0$. Hence there is some chance that $a'$ is chosen and there is some chance that $a$ is chosen, meaning $b$ is stochastic in state $s$. Thus the above conclusion follows from the definition of coverage when the policies are not identical under $s$.