Is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$?

Question

I'm learning the basics of RL and I'm struggling to understand the notion of terminal state in MDPs.

To ask my question straightforwardly: is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$? If I need to be more restrictive, assume a game setting, for example, chess.

My first hypothesis would be to define the terminal state as the state $s_T$ such that $p(s',r|s_T,a) = p(s',r|s_T)$, a state from which the transition is independent of the agent's actions. But that does not seem quite right. First, there is no particular reason why this state should be unique. Second, from this definition, it could also just be an intermittent state of "lag".

Answer 1

I don't know if there is a general definition of the terminal state based on the MDP transition probabilities.

But remember that we define our MDP problem in a $\mathbb{S}$ set of all possible states and a $\mathbb{A}(s)$ representing the set of all possible actions for each state. Based on that, probably there aren't any possible actions for the terminal state, so the transition probability $p(s', r |s_T, a)$ can be undefined. Based on that, in episodic tasks we need to define a the set of all possible terminal states $\mathbb{S}^+$ and distinguish this with $\mathbb{S}$.

Answer 2

As far as I remember, terminal state is a state from which agent cannot escape, i.e if the agent reached this state, he will never escape. In mathematical notation can be written as: $$ p(s^{'}, r|s_T,a) = \delta_{s^{'}s_T} \delta_{rr_{S_T}} $$ Where $\delta_{ab}$ is a Kronecker symbol, and by $r_{S_T}$ I mean the reward collected by the agent sitting in the terminal state from now till the end of the episode.

This state doesn't have to be unique. Imagine a Markov chain as a set of points, representing states, and arrows between the states with associated probabilities. Nothing prevents you from defining MDP where several nodes have only one outgoing arrow pointing to itself with the probability 1.