In the average reward setting we have:
$$r(\pi)\doteq \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{t=1}^{h}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$
$$r(\pi)\doteq \lim_{t\rightarrow\infty}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$
How is the second equation derived from the first?
We assume that our MDP is ergodic. Loosely speaking, this means that wherever the MDP starts (i.e. no matter which state we start in) or any actions the agent takes early on can only have a limited effect on the MDP and in the limit (as $t \rightarrow \infty$) the expectation of being in a given state depends only on the policy $\pi$ and the transition dynamics of the MDP.
This means that, eventually, $\mathbb{E}[R_t] = \mathbb{E}[R_{t+1}]$ for some large $t$. Therefore, as we take the average of our expected values of the rewards received for an infinitely long period of time, this will have converged due to what I just mentioned of $\mathbb{E}[R_t] = \mathbb{E}[R_{t+1}]$. To see why the two are equal, recall that the reward received is dependent on the current state and the action taken -- to better emphasise this I will briefly denote the reward at time step $t+1$ as $R(S_t, A_t)$. If we are in the steady state distribution, that is, the state distribution is now fixed, and our actions are still taken according to our policy, then the expected value of $R(S_t, A_t)$ will be the same for all future $t$ since neither the policy nor the state distribution are changing (recall that the average rewards are a way of evaluating a policy in the average-reward setting so for sure this does not change).
A way to think of this is that since we know that, eventually, $\mathbb{E}[R_t]$ will equal $\mathbb{E}[R_{t+1}]$, and so if we keep have an infinite number of these, the average of them will of course converge to the same value. Imagine if I gave you the sequence 1, 2, 3, 4, 4, 4, 4, ........, 4 and asked you to take the average - if we had an infinite amount of 4's then the average would of course be 4.
