How can a specific connectivity pattern be stored in an optimally compact representation?

Question

I am interested in optimizing the memory capacity of an AGI. Given a specific complex input an AI can create a simplified model. This is a problem that can be solved using sparse coding ^[1]. However, this solves only the problem of encoding and not the maintenance of online representations—in cognitive terms: the state of mind.

A default cognitive model of short-term memory can be separated in three different stages:

Encoding → Maintenance → Retrieval

One solution is to use specialized hardware ^[2][3], but I am interested in software approaches to this problem and I would thus like to emphasize that it is the digital representation, which I am most interested in.

With the exception of qubits, the smallest possible representation are binary digits. However, additional architecture is required to represent phase spaces (i.e. floating point precision memory) and higher-order representations maybe include arrays and dictionaries. (Optimizing these are trivial... or simply to be postponed until needed, according to Knuth).

• How can a specific connectivity pattern be stored in an optimally compact representation?
• Is there an implementation with concrete example code?
• What is the state-of-the-art?^*

_{* I will not specify "real-time" here, but the context is humanoid AGI.}

---

_{[1] Papyan, V., Romano, Y., & Elad, M. (2017). Convolutional Neural Networks Analyzed via Convolutional Sparse Coding. Journal on Machine Learning Research, 18(83): 1–52. arXiv:1607.08194 http://jmlr.org/papers/volume18/16-505/16-505.pdf}

_{[2] LeGallo et al. (2018). Mixed-precision in-memory computing. https://www.nature.com/articles/s41928-018-0054-8}

_{[3] IBM. (2018). IBM Scientists Demonstrate Mixed-Precision In-Memory Computing for the First Time; Hybrid Design for AI Hardware. https://www.ibm.com/blogs/research/2018/04/ibm-scientists-demonstrate-mixed-precision-in-memory-computing-for-the-first-time-hybrid-design-for-ai-hardware/}

Answer 1

Solution

In learning systems, the output of learning is a set of parameters. These parameters can be a tensor in the case of a CNN, a representation of a directed graph in the case of recursive network designs, or any other structure holding the results of training.

The general case of compactly storing the result of training involves four process elements. These elements can be combined into one algorithm if needed.

• Encoding of ordinals
• Lossless conversion of IEEE floats
• Serialization of the structure
• Lossless compression of the stream

Upon retrieval, the elements are reversed.

• Decompression of the stream
• Parsing to reassemble structure
• Expansion of IEEE float representations
• Decoding of ordinals

The encoding of ordinals requires the mapping of values to the minimal number of bytes or, most optimally, bits. The mapping must be reversible, meaning that there can be no information loss, so that decoding is completely reliable.

If the encoding and serialization packs the information tightly, there may be little gain with a general compression algorithm such as LZ4.

Notions of General Intelligence

Before devoting much time on developing humanoid AGI and modeling cognition and states of mind, it may be useful to curb expectations away from trendy ideas. There is much conflict between the naive conceptions of general intelligence and the realities of neurological systems in animals and people, which are far more sophisticated than implied in the trendy media hype. The most important considerations are these.

• The complexity of networks in the brain
• That there is a significant varieties of neurons in brains
• The impact of neuroplasticity
• Complexity of chemical signaling independent from nerves
• Whether intelligence is in fact multi-dimensional
• Large number of genes that correlate with intelligence
• The academic slant of concepts regarding intelligence
• Difference between brain signal topology and computing models
• Data-centricity

There are many articles on the web on these topics, along with critiques of concepts such as the g-factor and the idea of general problem solving and general intelligence. There are several questions here on this topic.

• Why did Artificial Intelligence projects fail?
• Can the IQ of an AI program be measured?
• Can a brain be intelligent without a body?
• Is the singularity concept mathematically flawed?
• Is there a ceiling to intelligence itself?
• Self-conscious artifical intelligence
• Is the Turing Test, or any of its variants, a reliable test of artificial intelligence?

There is no value in discouraging curiosity and serious research into understanding the nature of intelligence or efforts to expanding the boundaries of computing. The above comments are to assist in bringing the starting point of such inquiry into the realm of enthusiasm anchored by scientific methodology. Conjecture makes for great movie plots, but does not further science and does not lead to real world technology products and services.