I am interested in optimizing the memory capacity of an AGI. Given a specific complex input an AI can create a simplified model. This is a problem that can be solved using sparse coding [1]. However, this solves only the problem of encoding and not the maintenance of online representations—in cognitive terms: the state of mind.

A default cognitive model of short-term memory can be separated in three different stages:

Encoding → Maintenance → Retrieval

One solution is to use specialized hardware [2][3], but I am interested in software approaches to this problem and I would thus like to emphasize that it is the digital representation, which I am most interested in.

With the exception of qubits, the smallest possible representation are binary digits. However, additional architecture is required to represent phase spaces (i.e. floating point precision memory) and higher-order representations maybe include arrays and dictionaries. (Optimizing these are trivial... or simply to be postponed until needed, according to Knuth).

  • How can a specific connectivity pattern be stored in an optimally compact representation?
  • Is there an implementation with concrete example code?
  • What is the state-of-the-art?*

* I will not specify "real-time" here, but the context is humanoid AGI.

[1] Papyan, V., Romano, Y., & Elad, M. (2017). Convolutional Neural Networks Analyzed via Convolutional Sparse Coding. Journal on Machine Learning Research, 18(83): 1–52. arXiv:1607.08194 http://jmlr.org/papers/volume18/16-505/16-505.pdf

[2] LeGallo et al. (2018). Mixed-precision in-memory computing. https://www.nature.com/articles/s41928-018-0054-8

[3] IBM. (2018). IBM Scientists Demonstrate Mixed-Precision In-Memory Computing for the First Time; Hybrid Design for AI Hardware. https://www.ibm.com/blogs/research/2018/04/ibm-scientists-demonstrate-mixed-precision-in-memory-computing-for-the-first-time-hybrid-design-for-ai-hardware/

  • 1
    $\begingroup$ Welcome to SE:AI! I assume this is a hypothetical AGI we are discussing, but I feel like this is a deep encoding question involving binary representation. (Continuous learning also seems ancillary, but is good context for the problem:) I tweaked the title to reflect the central question. $\endgroup$ – DukeZhou Dec 20 '18 at 1:02
  • $\begingroup$ Thanks @DukeZhou! It's actually some theoretical groundwork for a basic AGI that I wish to implement IRL. Unfortunately, "deep encoding" doesn't seem to be a popular keyword. Thanks for pointing me to "Continuous learning". I think information-theory is applicable, but requires two Shannon-streams as the input destination is the temporary memory storage. Just pointing me to the right keywords is very helpful for my research. $\endgroup$ – noumenal Dec 20 '18 at 12:01


In learning systems, the output of learning is a set of parameters. These parameters can be a tensor in the case of a CNN, a representation of a directed graph in the case of recursive network designs, or any other structure holding the results of training.

The general case of compactly storing the result of training involves four process elements. These elements can be combined into one algorithm if needed.

  • Encoding of ordinals
  • Lossless conversion of IEEE floats
  • Serialization of the structure
  • Lossless compression of the stream

Upon retrieval, the elements are reversed.

  • Decompression of the stream
  • Parsing to reassemble structure
  • Expansion of IEEE float representations
  • Decoding of ordinals

The encoding of ordinals requires the mapping of values to the minimal number of bytes or, most optimally, bits. The mapping must be reversible, meaning that there can be no information loss, so that decoding is completely reliable.

If the encoding and serialization packs the information tightly, there may be little gain with a general compression algorithm such as LZ4.

Notions of General Intelligence

Before devoting much time on developing humanoid AGI and modeling cognition and states of mind, it may be useful to curb expectations away from trendy ideas. There is much conflict between the naive conceptions of general intelligence and the realities of neurological systems in animals and people, which are far more sophisticated than implied in the trendy media hype. The most important considerations are these.

  • The complexity of networks in the brain
  • That there is a significant varieties of neurons in brains
  • The impact of neuroplasticity
  • Complexity of chemical signaling independent from nerves
  • Whether intelligence is in fact multi-dimensional
  • Large number of genes that correlate with intelligence
  • The academic slant of concepts regarding intelligence
  • Difference between brain signal topology and computing models
  • Data-centricity

There are many articles on the web on these topics, along with critiques of concepts such as the g-factor and the idea of general problem solving and general intelligence. There are several questions here on this topic.

There is no value in discouraging curiosity and serious research into understanding the nature of intelligence or efforts to expanding the boundaries of computing. The above comments are to assist in bringing the starting point of such inquiry into the realm of enthusiasm anchored by scientific methodology. Conjecture makes for great movie plots, but does not further science and does not lead to real world technology products and services.

  • $\begingroup$ I am well aware of the caveats concerning the construct validity of intelligence, the nature of consciousness, and neurophysiological realism. I am currently focusing on aspects of implementation. What is possible with current software engineering and ML knowledge? For example, in your answer to "Why did [AI] projects fail?" you mention the importance of parallel over serial processing. However, when I read open code that solves specific problems I am concerned that those solutions are too slow and not suitable for AGI. There is a bias toward exact solutions and less emphasis on heuristics. $\endgroup$ – noumenal Dec 31 '18 at 10:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.