Is it appropriate to represent 'total failure' as an absorbing state?

Question

My understanding is that, in Markov decision processes, absorbing state are states which can transition only to themselves and that these transitions generate rewards of 0. I know that absorbing states are commonly used to represent goals, so an agent might get a non-zero reward when first entering the absorbing state, but all subsequent transitions generate 0 reward (effectively ending the episode).

My question is whether it is appropriate to also represent 'total failure' as an absorbing state. e.g. if the task is for Rabbit to return safely to his burrow (goal state which gives positive reward and ends the episode) while avoiding Fox (failure state which gives negative reward and ends the episode), would both burrow and fox-encounter be absorbing states? If not, how should fox-encounter be represented?

Answer 1

In an episodic problem, absorbing states are implemented to make the maths work similarly to continuing tasks. It allows one set of equations to cover two types of MDP (continuing and episodic). For example, the definition of state value might be

$$v(s) = \mathbb{E}[\sum_{k=0}^{\infty} \gamma^k R_{t + k + 1} | S_t = s]$$

This sum is poorly defined if $t$ cannot continue to infinity. Absorbing states fix that without changing the nature of the MDP significantly. Adding absorbing states is a form of mathematical generalisation that makes two otherwise different maths system behave identically from a theoretical standpoint.

So the short answer is: Yes, you should consider all terminal states to be absorbing if you are using that theoretical construct.

You should bear in mind that absorbing states do not usually represent anything physical. The absorbing states are only used to allow two sets of equations to be merged for convenience in the literature. In practical projects, e.g. if you implement code for an agent, absorbing states are not really something you need to care about. Instead you will end processing states/actions/rewards at any terminal state and start a new episode as necessary.