# Is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$?

I'm learning the basics of RL and I'm struggling to understand the notion of terminal state in MDPs.

To ask my question straightforwardly: is there a natural way to define the terminal state from the MDP transition probabilities $$p(s',r|s,a)$$? If I need to be more restrictive, assume a game setting, for example, chess.

My first hypothesis would be to define the terminal state as the state $$s_T$$ such that $$p(s',r|s_T,a) = p(s',r|s_T)$$, a state from which the transition is independent of the agent's actions. But that does not seem quite right. First, there is no particular reason why this state should be unique. Second, from this definition, it could also just be an intermittent state of "lag".

• Wait, are you asking how would the transition model be defined for a terminal state of an MDP? From the equations, that's what you seem to be trying to do, but that's somehow a different question than defining the "terminal state from the transition model" (which is your question in the body). So, I suggest that you edit this post to clarify what your question really is.
– nbro
Jan 28, 2021 at 12:16
• The terminal state is actually a case of a group of recurrent states in a MC except the group consists of only 1 state.
– user9947
Feb 27, 2021 at 14:05

But remember that we define our MDP problem in a $$\mathbb{S}$$ set of all possible states and a $$\mathbb{A}(s)$$ representing the set of all possible actions for each state. Based on that, probably there aren't any possible actions for the terminal state, so the transition probability $$p(s', r |s_T, a)$$ can be undefined. Based on that, in episodic tasks we need to define a the set of all possible terminal states $$\mathbb{S}^+$$ and distinguish this with $$\mathbb{S}$$.
As far as I remember, terminal state is a state from which agent cannot escape, i.e if the agent reached this state, he will never escape. In mathematical notation can be written as: $$p(s^{'}, r|s_T,a) = \delta_{s^{'}s_T} \delta_{rr_{S_T}}$$ Where $$\delta_{ab}$$ is a Kronecker symbol, and by $$r_{S_T}$$ I mean the reward collected by the agent sitting in the terminal state from now till the end of the episode.