What are options in reinforcement learning?

Question

According to a lecture (week 10) about Reinforcement Learning [1], the concept of an option allows searching the state space of an agent much faster. The lecture was hard to follow because many new terms were introduced in a short time. For me, the concept of an option sounds a bit like skills [2], which are used for describing high-level actions as well.

Are skills an improvement over options that includes the trajectory, or are both the same?

I'm asking for a certain reason. Normal deep reinforcement learning has the problem that the agent comes very often to a dead end, for example, in Montezuma's Revenge played at the Atari emulator. And the options framework promises to overcome the issue. But the concept sounds a bit too esoteric, and apart from the Nptel lecture, nobody else has explained the idea. So, is it useful at all?

Answer 1

An option is a generalization of the concept of action. The concept of an option (or macro-action) was introduced in the context of reinforcement learning in the paper Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning (1998) by Richard Sutton et al., so that to capture the idea that certain actions are composed of other sub-actions. Section 2 of the mentioned paper formally defines the concept of an option, which is a tuple composed of an initiation set, a policy, and a termination condition/set.

The authors of the mentioned paper give examples of options

Examples of options include picking up an object, going to lunch, and traveling to a distant city, as well as primitive actions such as muscle twitches and joint torques.

The option picking up an object, going to lunch, and traveling to a distant city is composed of other sub-actions (e.g. picking up an object), but is itself an action (or macro-action). A primitive action (e.g. joint torques) is itself an option.

A set of options defined over an MDP constitutes a semi-Markov decision process (SMDP), which are MDPs where the time between actions is not constant but it is variable. In other words, a semi-MDP (SMDP) is an extension of the concept of MDP that is used to deal with problems where there are actions of different levels of abstraction. For example, consider a footballer that needs to take a freekick. The action "take a freekick" involves a sequence of other actions, like "run towards the ball", "look at the wall", etc. The action "take a freekick" takes a variable number of time steps (which depends on the other sub-actions).

Semi-MDPs are thus used to deal with such problems that involve actions of different levels of abstraction. Hierarchical reinforcement learning (HRL) is a generalization (or extension) of reinforcement learning where the environment is modeled as a semi-MDP.

Curiously, certain models that have won the RoboCup (the famous AI football) context are based on the concept of semi-MDPs, options and HRL. See e.g. WrightEagleBASE, which use the MAXQ-OP (MAXQ online planning) algorithm.

Semi-MDPs can be converted to MDPs. The picture below (which is a screenshot of figure 1 of the mentioned paper that introduces the "options framework" in RL) illustrates the relationship between semi-MDPs and MDPs.

The empty circles (in the middle) are options, while the black circles (at the top) are primitive actions (which are themselves options).

In the paper Reinforcement learning of motor skills with policy gradients mentioned in the question, apparently, the term skill is not formally defined. However, I suppose that skills can be represented as options.