Reading the Retrace paper (Safe and efficient off-policy reinforcement learning) I saw they often use a matrix form of the Bellman operators, for example as in the picture below. How do we derive those forms? Could you point me to some reference in which the matter is explained?
I am familiar with the tabular RL framework, but I'm having trouble understanding the steps from operators to this matrix form. For example, why does $Q^{\pi} = (I -\gamma P^{\pi})^{-1}r$? I know that for the value $V$ we can write \begin{align} V = R + \gamma P^{\pi} V \\ V - \gamma P^{\pi} V = R \\ V (I -\gamma P^{\pi}) = R \\ V = R(I - \gamma P^{\pi})^{-1} \end{align} but this seems slightly different.
Picture from Safe and efficient off-policy reinforcement learning: