Can we define the AI singularity mathematically?

Question

The "AI Singularity" or "Technological Singularity" is a vague term that roughly seems to refer to the idea of:

1. Humans can design algorithms

2. Humans can improve algorithms

3. Eventually algorithms we design might end up being as good as humans at designing and improving algorithms

4. This might lead to these algorithms designing better versions of themselves, eventually becoming far more intelligent than humans. This improvement would continue to grow at an increasing rate until we reach a "singularity" where an AI is capable of making technological progress at a rate far faster than we could ever imagine

Also known as an Intelligence Explosion. This rough idea has been heavily debated as to its feasibility, how fast it'll take (if it does happen), etc.

However I'm not aware of any formal definitions of the concept of "singularity". Are there any? If not, do we have close approximations?

I have seen AIXI and the Gödel machine, but these both require some "reward signal" — it is unclear to me what reward signal one should choose to bring about a singularity, or really how those models are even relevant here. Because even if we had an oracle that can solve any formal problem given to it, it's unclear to me how we could use that to cause a singularity to happen (see this question for more discussion on that note).

Answer 1

I found someone that has done this thing! You can hear a good explanation in Marcus Hutter's answer to this question about rewards given to AIXI. He describes a work that seems to be referring to this paper:

Universal Knowledge-Seeking Agents for Stochastic Environments

I'll edit this answer later with a full explanation of the approach, but essentially the idea is that you use an AIXI model that does optimal reinforcement learning, giving it a reward that is based on information gained (phrased in a careful way to avoid a few common pitfalls). As a result, it learns to choose actions that give it the most information possible to predict the impacts of it's actions. This results in a "scientist" like behaviour, and you could imagine it doing things like turing the entire earth into a supercollider to better understand some physics laws if it decides that is the best approach for gaining maximum information. It would probably also do plenty of very unethical psycology experiements, for example, if it ended up deciding that human actions were important to predict and understand.

It's not a "safe" singularity in that sense, but that's okay, I didn't require that. It's at least a formal definition. It requires doing some uncomputable things, but the hope of future research is that we can make close enough approximations to those uncomputable things to be good enough anyway.

I feel this theory is lacking any explanation of how feasable such a task is since it uses uncomputable agents, so I won't accept it yet, but it's the best answer I've seen so far. And I'll be watching future research closely to see if they can get a better handle on feasability, there seems to be quite a bit of work that has gone on in finding computable approximations to AIXI. The reason I care about feasability is because it is very relevant for mathmatically deciding how plausable something like an "intelligence explosion" actually is. So if a theory doesn't talk about feasability, it is missing out on a big piece of this question. Still this theory seems hopeful. For example, maybe there are fundamental computational limits to maximizing some reward functions, and we can prove that even certain levels of approximations for this reward function aren't computable. That would be a really interesting negative result.

In general, I think the idea of using reinforcement learning, then choosing a reward function that tries to capture something instrinsic (such as "curiosity") is a very good approach to trying to formally define the singularity. I look forward to seeing other potential reward functions defined in the future, I don't expect this to be the only one.

Answer 2

Here is one idea. I'll start with a more specific "mathematical singularity", defined as an algorithm that can do the following in N hours or less (for all $N >= 1$):

1. State equivalent versions (up to notional differences) of all mathematical theorems/conjectures that humans will read and understand in N*20 years after 2018 that can be stated formally in Metamath (this in an arbitrary choice, but Metamath is general enough to include quantum logic and extensions of ZFC so it seems like a decent place to start with. Feel free to instead use Coq, Isabelle, Lean, etc. instead if you prefer), assuming those humans never have access to a "mathematical singularity" capable algorithm and their mathematical community continues living and functioning intellectually in a manner similar in capacity to how it did in 2018
2. Of those problems, provide correct proofs (these may not be readable, that's ok) of all of those that will be solved by those humans in N*20 years.

This of course does not fully capture all mathematical progress that humans will make in those years: a big component missing is "readable proofs" and concepts that can't be captured in metamath. But it is something that is theoretically formal.

I know that this doesn't include any "continual improvement", what I am referring to here is simply a threshold such that when an algorithm passes it, I think it is sufficently powerful enough to be considered as "intelligent enough" that it has reached close to singularity levels of intelligence. Feel free to adjust the (20 years) constant in your head to match your preferred threshold.

I'm not going to accept this answer because it is lacking "continual improvement", but I brought it up because if we can't figure out how to define it mathematically, perhaps simply having "sufficient criteria" in various domains could be a good start.

Edit: I suppose that the singularity typically involves an assumption of the development of an intelligence that is superior to human society. This implies that it is capable of at least doing the things that our society does, so there is probably a good argument to be made here that "proof accessibility" and "method teachability" are vital to this problem.

I mean, if we think of the current state of the field of calculus, it has gone from an arcane topic only understood by a few field experts, to now being readily accessible and teachable to high school students. While that didn't require proving any new major mathematical theorems, one could argue that much of our technological progress didn't come until advanced mathematical machinery developed (calculus) became accessible to a wide range of people.

I was going to make an argument about how "the difference is that computers can learn quicker: they can read through massive proofs very quickly". But I suppose that depends on the architecture of whatever kind of "thing" is achieving the singularity. I.e., here is a (non-exhaustive) list two possible outcomes:

• There is only one "mind" that is achieving all of this. In that case, that mind has all the knowledge it needs and it doesn't need to teach anyone to progress further, so this point is sorta irrelevant. However, I can still see an argument for "teachability" if we want to utilize this vast amount of knowledge the AI has gained in human society, if possible.
• There is a simulated "society" of virtual minds that are interacting with each other, that, together, achieve the mathematical singularity. If a single "mind" in this "society" isn't able to easily use and understand the work done by another mind, then the point of "teachability" is very important to prevent individual minds from having to continually recreate the wheel, so to speak.

Without our biological limitations these digital minds may have very different "teaching" methods, but I think here is the ideal additional requirement for a "mathematical singularity":

3. These proofs must be (eventually, perhaps not until spending quite a bit of time) accessible to a graduate mathematician, via proving pdf textbooks (or other similar teaching materials) that cover the same material that human mathematical textbooks would have covered after N*20 years in a way that is accessible to the typical graduate mathematician.

However we have now lost some formality in this: textbooks usually contain lots of exposition and analogies that are difficult to formally measure and may not even be relevant for the AI. Here is an alternate option that is not as good, but still close:

3. The algorithm must present its results in a form that can be used by any other algorithm that also can achieve the "mathematical singularity" to "skip ahead" to N*20 years, and then immediately continue progress from there.

However this criteria has a trivial exploit: an algorithm might as well just provide a 'save state' and a 'program' to run that save state. Conceivably any algorithm that can achieve the mathematical singularity is at least capable of executing code, so providing a 'save state' and 'program' passes this criteria without making it at all accessible (The caveat here is if it uses some sort of model of computation that requires special hardware such as quantum computing or black hole computing to prevent slowdown, but that's besides the point)

I think I prefer this alternative:

3. These proofs must be similar in length as the (formalized versions of) proofs the human academic community would have made in those 20*N years

"length" is tricky here: it is possible to prove a very difficult theorem very succinctly by simply referencing a very powerful lemma. But here is one example metric:

$$length(Proof) = lengthInSymbols(Proof)+\sum_{symbol \in Proof} \frac{length(symbol)}{numberOfTimesUsedInOtherProofs(symbol)}$$

Where "Other Proofs" is the set of all proofs read and understood by humans in those N*20 years, and "symbols" refers to things such as "Green's Theorem" or "$\in$". Hopefully the idea is apparent here: if something is used frequently in many proofs, it is a "common technique" that isn't vital to that proof, and thus doesn't contribute as much to the "length" of that proof. Finding a potentially more suitable metric here seems like a much more tractable problem then defining the mathematical singularity itself and I suspect this is studied elsewhere more, so I'll leave it at this for now.

Answer 3

I would approach it from a direction different from @Phylliida, though there seems to be nothing wrong with her answer.

IMHO, when AI

• A: is sufficiently general,
• B: is able to direct its own evolution,
• C: has control (whether direct or indirect) of all the resources needed to grow and evolve, and
• D: has the goal of growing and evolving to solve a big, important problem,

then the "singularity" will have been reached.

"sufficiently general" means that its growth is not limited by the code that initially defines it: it can re-write its own code (through "offspring in a sandbox", or directly; doesn't matter which).

Genetic programming is currently clumsy but is indeed sufficiently general.

C is something that would most likely need to be given to it, so nobody will be able to unplug it.

D is easy. I'd like to see an effort in the Futurist and AI communities to choose such goals.

This isn't a mathematical definition; it's something more mundane; but I think it's to the point.

Answer 4

I believe that mathematical theorems are social constructions which are formalised by virtue of rigorous proofs facilitated by an academic peer review process; in other words, I am not a mathematical Platonist. You ask: “Can we define the AI singularity mathematically?” I personally see no reason why the so-called AI singularity cannot be defined in mathematical terms. Let us pretend that a gifted mathematician is able to provide a convincing and logically consistent set of rigorous proofs for a mathematical conjecture pertaining to the AI singularity. The aforementioned mathematician then submits the paper to a highly prestigious mathematics journal, such as Journal of the American Mathematical Society. If this paper passes the stringent academic peer review process for publication, then this would constitute peer acceptance that the AI singularity can indeed be defined in mathematical terms.