2
$\begingroup$

My understanding is that, in Markov decision processes, absorbing state are states which can transition only to themselves and that these transitions generate rewards of 0. I know that absorbing states are commonly used to represent goals, so an agent might get a non-zero reward when first entering the absorbing state, but all subsequent transitions generate 0 reward (effectively ending the episode).

My question is whether it is appropriate to also represent 'total failure' as an absorbing state. e.g. if the task is for Rabbit to return safely to his burrow (goal state which gives positive reward and ends the episode) while avoiding Fox (failure state which gives negative reward and ends the episode), would both burrow and fox-encounter be absorbing states? If not, how should fox-encounter be represented?

$\endgroup$

1 Answer 1

3
$\begingroup$

In an episodic problem, absorbing states are implemented to make the maths work similarly to continuing tasks. It allows one set of equations to cover two types of MDP (continuing and episodic). For example, the definition of state value might be

$$v(s) = \mathbb{E}[\sum_{k=0}^{\infty} \gamma^k R_{t + k + 1} | S_t = s]$$

This sum is poorly defined if $t$ cannot continue to infinity. Absorbing states fix that without changing the nature of the MDP significantly. Adding absorbing states is a form of mathematical generalisation that makes two otherwise different maths system behave identically from a theoretical standpoint.

So the short answer is: Yes, you should consider all terminal states to be absorbing if you are using that theoretical construct.

You should bear in mind that absorbing states do not usually represent anything physical. The absorbing states are only used to allow two sets of equations to be merged for convenience in the literature. In practical projects, e.g. if you implement code for an agent, absorbing states are not really something you need to care about. Instead you will end processing states/actions/rewards at any terminal state and start a new episode as necessary.

$\endgroup$
2
  • $\begingroup$ The OP states "absorbing states are commonly used to represent goals", but is that really the case? Or is the absorbing state a state where the agent (conveniently) transitions to after having reached the goal? See, for example, this answer. It might be worth clarifying this. $\endgroup$
    – nbro
    May 15 at 15:36
  • $\begingroup$ @nbro: Yes I read the "used to represent goals" as "used for episode completion states associated with high rewards". I'm not sure if that slightly wooly shorthand needs correcting here, as it was clear to me what the OP was asking about (i.e. do we add absorbing states when termination reward is low as well as when it is high?) $\endgroup$ May 15 at 16:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .