A benchmark comparison of systems comprised of separately trained networks relative to single deeper networks would not likely reveal a universally applicable best choice.1 We can see in the literature the increase in the number of larger systems where several artificial networks are combined, along with other types of components. It is to be expected. Modularization as systems grow in complexity and demands on performance and capability grow is as old as industrialization.
Our laboratory works with robotic control, thermodynamic instrumentation, and data analysis, artificial networks are components in these larger system contexts. we have no single MLPs or RNNs that by themselves perform any useful function.
Contrary to the conjecture about hierarchies decades ago, the topology approach that seems to work well in most cases follows the more common system module relationships that are seen in power plants, automated factories, aeronautics, enterprise information architectures, and other complex engineered creations. The connections are those of flow, and if those are designed well, oversight functions are minimal. Flow occurs between modules involving protocols for communications, and each module performs its function well, encapsulating the lower level of complexity and functional detail. It is not one network overseeing another that seems to emerge most effective in actual practice but balance and symbiosis. Identification of clear master-slave design in the human brain seems to be equally slippery. It may be naive to assume that some all knowing subsystem can be a single causal factor in the remarkable collaborative synchronization of a number of other subsystems.
The challenge is not finding the information paths that make up the system information topology. The information flow is often obvious upon problem analysis. The difficulty is in discovering the best of strategies to train these independent networks. Training dependencies are common and often critical, whereas in animals, the training occurs in situ or not at all. We're discovering conditions under which that kind of learning in our systems is practical and how to achieve it. Most of our research along these lines is intended to discover ways to achieve higher reliability and lower burden in terms of research hours to get it.
Higher functionality is not always of benefit. It often produces lower reliability and consumes additional development resources with little return. Find a way to marry higher level automation, resource thrift, and reliability into one development process, and you might win an award and honorable mention around the web.
Parallel systems that have the same objective is a good idea, but not a new one. In one aeronautics system, nine parallel systems have the same objective, in groups of three. Each group uses a different computing approach. If two of the systems using the same approach provide the same output and the third differs, the matching output is used and the difference in the third is reported as a system fault. If two of the different approaches provide similar results and the third differs substantially, a merge of the two similar results is used and the third is reported as a use case to further develop the dissenting approach.
The improved fault tolerance has a cost, eight more systems and associated computing resources and connectivity plus the comparators at the tail, but in systems that are a matter of life and death, the extra costs are paid and the reliability is maximized.
Dynamic topological adaptation is related to redundant systems and fault tolerance but in some ways is quite distinct. In that area of development, the technology to follow is neuromorphic computing, which is partly inspired by neuroplasticity.
One last distinction to consider is between process topology, data topology, and hardware topology. These three geometric frames can produce greater efficiency together if aligned in specific ways that produce more direct mappings between the relationships between flow, representation, and mechanics. They are, however, distinct topologies. The meaning of alignment may not be apparent without diving deeply into these concepts and the details that emerge for specific product or service objectives.
Footnotes
[1] Deep networks that are trained as a single unit and function without connectivity to other artificial networks are not necessarily monolithic. Most practical deep networks have heterogeneous sequence of layers in terms of their activation functions and often their cell types.
