3
$\begingroup$

In Sutton & Barto's Reinforcement Learning: An Introduction, page 54, the authors define the terminal state as following:

Each episode ends in a special state called the terminal state

But the authors also say:

the episodes can all be considered to end in the same terminal state, with different rewards for the different outcomes. Tasks with episodes of this kind are called episodic tasks.

I believe there is also a fundamental difference between a terminal state, nonterminal states and plain, normal states:

In episodic tasks we sometimes need to distinguish the set of all nonterminal states, denoted S, from the set of all states plus the terminal state, denoted S+.

In the first quote, it appears as if the terminal state is just a term to describe the final state of an episode, but, from the second quote, I understand that the terminal state is the same no matter the outcome of the episode. If we consider the game of chess, what would we consider as a terminal state? Would it be the state that, if reached, will end the game (checkmate), no matter the result (win, loss)? But then how can we describe a state that would lead to draw? If we say about a state that leads to a draw that it's a nonterminal state since we can play an "infinite" number of turns without reaching a win or a loss hence without reaching the terminal state, aren't we implicitly supposing that reaching a draw isn't a result for which we should attribute a reward (e.g. 0)? And if we name a state that leads to a draw a terminal state, then what would be the difference between a normal state and a nonterminal state?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
3
$\begingroup$

Terminal state is always the same in the sense that it represents the same thing, that the episode is over. They don’t need to be the exact same state; for instance you could have an $n$ by $n$ grid world where the top right and bottom left states are terminal as when you reach these your agent dies. These are both terminal but not the same state.

For chess it would be any state that when reached the game ends (regardless of win/draw/lose). The difference between these terminal states is what reward you will receive for reaching it.

Finally, normal states are non-terminal states, so there is no difference.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ It may also be a good idea to describe what an absorbing state is, although this was not part of the question, but it's related. $\endgroup$
    – nbro
    Jun 10, 2021 at 10:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .