# Finding goals in Hierarchical Reinforcement Learning

In a recent paper Data-Efficient Hierarchical Reinforcement Learning, O Nachum, S Gu, H Lee, S Levine, 2018, a promising agent controlling technique called Hierarchical Reinforcement Learning was introduced. It is some kind of layered policy for controlling ant-like robots in a maze.

For example, the main controller is able to run sub-controllers move and push, and this allows the ant to move to a goal, even an obstacle is on the way.

But there is something in the paper which i didn't understand: How to find the goals. According to the paper, the lowlevel actions “move” and “push” are equal to goals. And these goals have to be inferred from demonstrations. In the paper, they write:

For generality, we develop a scheme where lower-level controllers are supervised with goals that are learned and proposed automatically by the higher-level controllers.

How exactly is the matching between the observation and the goal state done?

Meaning of Low Level Goals in Data-Efficient Hierarchical Reinforcement Learning *

* Ofir Nachum, Shixiang Gu, Honglak Lee, Sergey Levine

Before explaining why a move or a push are goals, let's examine a statement from a prior paper. Although Sergy Levine (Berkeley) is well published and a respected contributor, we can see why Ofir Nachum (Google) is named first on this particular paper if we review Bridging the Gap Between Value and Policy Based Reinforcement Learning **

** Ofir Nachum, Mohammad Norouzi, Kelvin Xu1, Dale Schuurmans

This earlier paper provides the basis for using a single algorithm for both higher goals and lower goals. Note the statement at the end of the Introduction section: "... the actor and critic can be unified in a single model that coherently fulfills both roles."

If you read the abstract and consider the permutations of the 3x2 matrix referenced in the above quote, a clearer perspective on hierarchical reinforcement can be gained.

• Higher Policy Network
• actor-critic model
• actor
• critic
• Lower Softmax Network
• actor-critic model
• actor
• critic

What is implied in the paper's abstract and the above introductory remark (detailed in the summations and exponentials further down in the paper) is that not only does the policy layer drive the learning process of the learning of the lower level network employing softmax as an activation function, but the critic can be unified, "... eliminating the need for a separate critic." This means that a single feedback vector of signals can be used to drive the learning at two layers.

That makes the system more data-efficient. It is not difficult to imagine why. A single database of experience is required instead of two.

Much of the work being done in AI today is building toward a hierarchical network model where a policy is learned in parallel (in the time domain) with the learning within the network capable of executing the policy. This parallelism across levels of process not only drives toward the AI goal of layering in two dimensions ...

• From input to output AND
• From higher level goals to lower level ones

... but brings the network architecture closer to that of mammilian brains where neurotransmitters can act as critics for multiple layers and organs of instinctual and motor operations simultaneously. Whether this leads eventually to producing the learning of logic and rational thought, therefore modelling cognitive functions, is yet to be seen.

Robotic Implications

Consider an expanded set of operational layers, each of which begins with basic capabilities, each of which cannot be initially constructed to be optimized for all in sito scenarios (real life system internal scenarios), and thus each of which must learn in concert.

1. What the robot owner wants it to learn and do
2. What the robot's learning system must learn as policies to do it in a variety of circumstances
3. What operations must be performed to execute a wide range of policies
4. What movements must be performed for each control channel (corresponding to the robot's degrees of motion)
5. What step edges must be sent to the stepper motor controllers while maintaining what states of direction (clockwise or counterclockwise)

In the context of the abstract, "Move" and "Push", are examples of level 3 in the above list.

Consider a somewhat larger segment of the abstract:

In this paper, we study how we can develop HRL algorithms that are general, in that they do not make onerous additional assumptions beyond standard RL algorithms, and efficient, in the sense that they can be used with modest numbers of interaction samples, making them suitable for real-world problems such as robotic control. For generality, we develop a scheme where lower-level controllers are supervised with goals that are learned and proposed automatically by the higher-level controllers. To address efficiency, we propose to use off-policy experience for both higher and lower-level training.

(An application of the strategy and mathematical detail could presumably be used to write an embedded program for a robot that can teach it to vacuum your living room. This is obviously a technology possibility that opens the door to thousands of productization opportunities.)

To understand what off-policy experience means, it is clearly described in Q(λ) with Off-Policy Corrections.

This additional paper explains it from a robotic perspective and produces a good bridge to the paper referenced in this question: Off-policy experience retention for deep actor-critic learning

Additional Response to the Q Author's 1st Comment Below

Perhaps in that paper and although your thinking is good, that's not quite the general case. When you say, "Actor has its own neural network, and the critic has it's own neural network," that is not thinking recursively. Think of it this way, "The actor of one hierarchical layer IS the critic of another"

It may help to remind you what you already know from your knowledge of words and may have studied in some mathematical detail already.

The following concepts overlap in meaning considerably, even though there are fine shades of difference in context and mathematical expression.

• CONVERGED vs DIVERGED
• CORRECTNESS vs ERROR
• PROXIMITY vs DISPARITY
• LOW or HIGH SUM in PID controllers
• REWARD vs PUNISHMENT
• AWARD vs PENALTY
• PROFIT vs LOSS
• POSITIVE and NEGATIVE CRITIQUE
• PLEASURE vs PAIN
• WELLNESS vs DIS-EASE

All these terms and term pairs are approximations of the quality of some entity's behavior with respect to some ideal at some point of time. They only differ in the common context in which they are use and the implied specific intention of the agent providing the information to the entity to control, encourage, or teach the idea behavior.

For such concepts to be effectively applied in a learning system, the concept must be quantified and the direction of the metric with respect to the underlying concept must be consistent. By this is meant that each floating point or ordinal value must be ordered in the same direction throughout the range of the measurement. (You can't have 1.7 mean excellent, 4.2 mean average quality, and 2.9 mean poor. Such would create an ambiguity in the feedback signalling.

The term Feedback Signalling is appropriate in the Cybernetic mathematics domain because that's what every item in the above list is. The word Signalling is more accurate than the word Signal because there may be more dimensions than one to the feedback — a vector, matrix, cube, or hyper-cube rather than a scalar.

When you say, "The actor rewards the low level ... controller," that is not necessarily the case. Whether some reinforcement signal passes from the higher levels to the lower levels in addition to a signal more analogous to a command does not seem to be the primary feature of the theory Ofir Nachum's is documenting in the above papers.

Keep in mind that the output of any program, neural net or not, into another program can simply be the output of one into the input of another. Various researchers are theorizing about and experimenting with differing connectivity architectures. The output of a higher layer neural net could simply be part of the input vector of the next layer below.

The key to the research in the papers discussed is whether two or more layers can share the same reinforcement data and algorithm. This is probably of rising interest because of the high demands recursive layering of deep, wide, and multidimensional neural networks place on CPU cycles, local memory, persisted data synchronization and access throughput and response time, and network resources.

The removal of redundancy in the overall architecture is directly related to cost, equipment size, and speed and accuracy of reaction.

• If I understand the actor-critic-model in “Learning to Play Donkey Kong Using Neural Networks and Reinforcement Learning, 2017” right (a conference paper, published by Springer), then the actor has its own neural network, and the critic has it's own neural network. The actor rewards the lowlevel “push” controller. That means, the subcontroller is no longer rewarded by the human-demonstration but internally by the actor. – Manuel Rodriguez Jun 23 '18 at 18:00