I read some light material earlier about the possibility of building a hierarchy of agents, where the agents at the leaves solve primitive tasks while higher-level agents are optimized for orchestrating direct-descendant agents.
According to my understanding, each agent would have different objective functions and possible moves.
How can I design such a hierarchy of agents each of which with different goals? Are there any examples?
In the context of reinforcement learning, the idea of modeling your goal-oriented problem as a hierarchy of multiple sub-problems is called hierarchical reinforcement learning, which gives rise to concepts such as semi-Markov decision processes and options (aka macro actions). The article The Promise of Hierarchical Reinforcement Learning presents and describes the topic quite well, so I suggest you read it.
However, this idea of solving multiple sub-tasks in order to solve a bigger task isn't limited to RL. For example, in the paper Neural Programmer-Interpreters (NPIs), without referring to the traditional RL topics, a model is proposed to write programs by composing simpler ones.
Another example is genetic programming with automatically defined functions (ADFs), where sub-programs (the ADFs) can be re-used in different parts of the program if that's beneficial according to the fitness.
So, there are different ways you can design a system to solve a big task by solving multiple sub-tasks and then compose them. The approach that you choose depends on your use case. If you want to build programs, then NPIs can be a start. If you want to incorporate a time component in your system, then HRL is probably a viable approach. There are several HRL algorithms. Some of them (e.g. MAXQ-OP) have been successfully used to solve the RoboCup challenge.
