Establishing Names and Terms
The mathematician's name is Kurt Gödel, an Austrian-American, and the machine concept named after him is the Gödel machine. The idea of meta-learning (learning how to learn) can be extended recursively to learning how to learn how to learn how to learn and so on.
Taking a decision is when the decision is external to the system and the system either copes or executes that decision, whereas making a decision is when the decision is internal to the system and the system executes it, reports it, or delegates it.
Both of these have been tried in the laboratory or in real systems and various levels of recursion have achieved success, but not indefinite recursion.
A Few Points from AI History
Creating a learn how to learn recursion was the object of several early LISP efforts at MIT, and meta-learning is in active research in multiple universities and corporations as of this writing. The first successful application of a single level of recursion were the expert systems that could acquire knowledge using meta-rules, which were rules about creating new rules.
Current Artificial Networks Regarding Meta-learning
It is correct that back propagation is not meta-learning. It is a single level of learning in the above recursive learning concept. Artificial networks have been applied to learn to adjust the hyper-parameters of another network, but these strategies require large quantities of data, and they don't cognate.
They don't know how to adjust hyper-parameters. They don't build models of learning and then use those models to learn. The adjustment mechanism is the application of a learned function, not a conception of learning.
Requirements for Meta-learning
To establish a meta-learning paradigm, we must consider the elements of learning.
- Existence of function abstraction
- Ability to guess a parameterized function
- Ability to mutate that function
- Ability to tune the parameters
- A target functional behavior
- A way to detect whether the function is approaching the target behavior
- A strategy for deciding when to mutate and when to tune
- A way of performing experiments in isolation
- A way to recurse in a way that incorporates ALL of the above
Once this has been achieved, it may be described as self-improvement and may qualify as what most theorists think of as a Gödel machine.
Specifics in the Question
Can [an artificial] network [make] decision about its own weights (update of weights) during training phase or during the phase of parallel training and inference?
Yes.
[Can] one [level] of [a] hierarchical [artificial network] [make] decision[s] about weights of other [levels]?
Yes, but in a limited way as of the time of this writing.
I am very keen to understand about self-awareness ...
Self-awareness is the ability of the system to analyze and either utilize the result of that analysis, report it, or delegate based on the result IN COMBINATION WITH the ability to use itself as the object of analysis.
This can be as simple as a program that parses Java and produces statistics using its own code as the code to parse. It can be as complex as an anthropologist studying humanity. It can be as personal as a human looking in the mirror and wondering about the qualities of person they are. It can be as deep as someone wondering their purpose in their current life.
In all cases, there are those two qualities.
- The ability to perform a type of analysis that could be applied toward a class of objects of which itself is a member
- The selection of itself for analysis
There is one other aspect of awareness beyond these two that are not required but usually associated with awareness: Some way of directing attention of the analysis capability to itself regularly. Here are a few examples from human experience.
- Practice of daily meditation
- Reading a book on purpose
- Seeing a cognitive behavioral therapist
- Writing an autobiography
- The first, fourth, and tenth Step of anonymous programs
- Watching a candid movie of one's self
- A family member that likes to step back and evaluate the family
- Religious accountability groups
Self learning is like self-awareness except that learning replaces awareness. Obviously, self-learning is dependent on some degree of self-awareness. We now have a more activity based list because of the additional element of adjustment or inception of action.
- Decide, on the basis of personal abilities and interests, after evaluating options for living, to relocate
- Realizing that self cannot remember commitments to meetings to buy a technology device that notifies the user before such commitments and using the calendar app rigorously
- Finding that repeated attempts to eat greens rather than junk food is not working and joining a community of those with eating disorders with solutions to that issue
- Deciding that childhood views of future were good ones and abandoning current paths for ones that were bolder and will likely lead to fulfillment
Self-improvement is a super-set of self-learning, since the only point of learning is improvement, however there are forms of improvement that don't strictly involve learning. A system (or person) can execute something already learned that results in an improvement.
Artificial general intelligence (e.g. Gödel machine)
There is no mathematical proof that a Gödel machine would exhibit general artificial intelligence. More importantly, there is no proof and a considerably body of evidence against the proposition that humans are generally intelligent.
Lastly, there is no proof that general intelligence is achievable in a biological or artificial system. Gödel's second incompleteness theorem is strong evidence that such ideas may be naive.
[Artificial] networks are usually mentioned as examples of special, single-purpose intelligence, but I can not see the reason for such limitation if [they] essentially [try] to mimic human brains, at least in purpose if not in mechanics.
Artificial networks emerged out of a desire to mimic human brains, but the perceptron design is based on an old view of neuron functionality. Simulating a single neuron might require an entire rack of CPUs and GPUs.
Furthermore, we are not yet clear on what intelligence is. Definitions proposed vary widely. The fields of artificial networks, bioinformatics, ontology of ideas, semantics, and the relationships between stable adaptivity and algorithms are all in their early stages, both theoretically and experimentally.
Maybe this desired [capability] is already effectively achieved [or currently emerging] in the operation of recurrent ANNs as the effect of collective behavior?
No, but RNNs are a tiny step closer to biological neurons in that their layers maintain state beyond the attenuation matrix.
By attenuation matrix is meant the matrix of parameters used in vector-matrix multiplication to control the strength of signals from the activation functions of one layer to the activation functions of the next. It's common name in machine learning literature is simply, "The parameters." Learning occurs as they change to converge the network on an optimal state.
RNNs are also capable of being Turing complete, so they can theoretically realize arbitrary algorithms.
However, the Requirements for Meta-learning above are not an expected capability of any single instance of an artificial networks, either of the MLP (multilayer perceptron) type or the RNN type. Whether balancing two networks in symbiotic arrangement as in GANs, or a comprehensive recursive algorithm, or the simulation of neural and synaptic plasticity in silicon will lead to Gödel machines is unknown.
Let Science be Science
Whether ideas such as general intelligence are possible is unknown. Whether ideas about singularities in books and mass media are realistic is unknown. Whether the prophetic warnings of screenwriters about the emergence of artificial entities that become the dominant species on earth is unknown.
Jaques Ellul is perhaps the most scientifically accurate prophetic voice. In his Techological Sociey, he presents heaps evidence in support of the idea that humans are already serving an autonomous technology and have been since prior to industrialization.
A Swiss philosopher, Francis Schaeffer, once prophecied, "They will say all kinds of things in the name of science that have nothing to do with science." We should be careful to keep conjecture from reaching the status of scientific fact in technical conversation. If we have no carefully drawn theory or empirical evidence to support a conjecture, it should be stated as a proposal, not a conclusion.
