1
$\begingroup$

I'm learning the basics of RL and I'm struggling to understand the notion of terminal state in MDPs.

To ask my question straightforwardly: is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$? If I need to be more restrictive, assume a game setting, for example, chess.

My first hypothesis would be to define the terminal state as the state $s_T$ such that $p(s',r|s_T,a) = p(s',r|s_T)$, a state from which the transition is independent of the agent's actions. But that does not seem quite right. First, there is no particular reason why this state should be unique. Second, from this definition, it could also just be an intermittent state of "lag".

$\endgroup$
2
  • $\begingroup$ Wait, are you asking how would the transition model be defined for a terminal state of an MDP? From the equations, that's what you seem to be trying to do, but that's somehow a different question than defining the "terminal state from the transition model" (which is your question in the body). So, I suggest that you edit this post to clarify what your question really is. $\endgroup$ – nbro Jan 28 at 12:16
  • $\begingroup$ The terminal state is actually a case of a group of recurrent states in a MC except the group consists of only 1 state. $\endgroup$ – user9947 Feb 27 at 14:05
1
$\begingroup$

I don't know if there is a general definition of the terminal state based on the MDP transition probabilities.

But remember that we define our MDP problem in a $\mathbb{S}$ set of all possible states and a $\mathbb{A}(s)$ representing the set of all possible actions for each state. Based on that, probably there aren't any possible actions for the terminal state, so the transition probability $p(s', r |s_T, a)$ can be undefined. Based on that, in episodic tasks we need to define a the set of all possible terminal states $\mathbb{S}^+$ and distinguish this with $\mathbb{S}$.

$\endgroup$
1
$\begingroup$

As far as I remember, terminal state is a state from which agent cannot escape, i.e if the agent reached this state, he will never escape. In mathematical notation can be written as: $$ p(s^{'}, r|s_T,a) = \delta_{s^{'}s_T} \delta_{rr_{S_T}} $$ Where $\delta_{ab}$ is a Kronecker symbol, and by $r_{S_T}$ I mean the reward collected by the agent sitting in the terminal state from now till the end of the episode.

This state doesn't have to be unique. Imagine a Markov chain as a set of points, representing states, and arrows between the states with associated probabilities. Nothing prevents you from defining MDP where several nodes have only one outgoing arrow pointing to itself with the probability 1.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.