I read some light material earlier about the possibility of building a hierarchy of agents, where the agents at the leaves solve primitive tasks while higher-level agents are optimized for orchestrating direct-descendant agents.

According to my understanding, each agent would have different objective functions and possible moves.

How can I design such a hierarchy of agents each of which with different goals? Are there any examples?


In the context of reinforcement learning, the idea of modeling your goal-oriented problem as a hierarchy of multiple sub-problems is called hierarchical reinforcement learning, which gives rise to concepts such as semi-Markov decision processes and options (aka macro actions). The article The Promise of Hierarchical Reinforcement Learning presents and describes the topic quite well, so I suggest you read it.

However, this idea of solving multiple sub-tasks in order to solve a bigger task isn't limited to RL. For example, in the paper Neural Programmer-Interpreters (NPIs), without referring to the traditional RL topics, a model is proposed to write programs by composing simpler ones.

Another example is genetic programming with automatically defined functions (ADFs), where sub-programs (the ADFs) can be re-used in different parts of the program if that's beneficial according to the fitness.

So, there are different ways you can design a system to solve a big task by solving multiple sub-tasks and then compose them. The approach that you choose depends on your use case. If you want to build programs, then NPIs can be a start. If you want to incorporate a time component in your system, then HRL is probably a viable approach. There are several HRL algorithms. Some of them (e.g. MAXQ-OP) have been successfully used to solve the RoboCup challenge.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.