The "AI Singularity" or "Technological Singularity" is a vague term that roughly seems to refer to the idea of:

  1. Humans can design algorithms

  2. Humans can improve algorithms

  3. Eventually algorithms we design might end up being as good as humans at designing and improving algorithms

  4. This might lead to these algorithms designing better versions of themselves, eventually becoming far more intelligent than humans. This improvement would continue to grow at an increasing rate until we reach a "singularity" where an AI is capable of making technological progress at a rate far faster than we could ever imagine

Also known as an Intelligence Explosion. This rough idea has been heavily debated as to its feasibility, how fast it'll take (if it does happen), etc.

However I'm not aware of any formal definitions of the concept of "singularity". Are there any? If not, do we have close approximations?

I have seen AIXI and the Gödel machine, but these both require some "reward signal" — it is unclear to me what reward signal one should choose to bring about a singularity, or really how those models are even relevant here. Because even if we had an oracle that can solve any formal problem given to it, it's unclear to me how we could use that to cause a singularity to happen (see this question for more discussion on that note).

  • $\begingroup$ Is the AI singularity a problem or not (where by "problem" I mean something that has a solution)? Or is the AI singularity only a "situation"? $\endgroup$ – nbro Feb 17 at 23:37
  • 3
    $\begingroup$ Related - ai.stackexchange.com/questions/7337/… - you can see from my answer that I do not believe that there is any semi-rigorous maths formulation of "The Singularity". If there was one, I am sure Kurzweil would have found and promoted it . . . $\endgroup$ – Neil Slater Feb 17 at 23:51
  • 1
    $\begingroup$ I believe it would be possible via set theory/complexity theory. But, no matter how rigorous the formula, it would still be only a hypothesis like the the Drake equation. $\endgroup$ – DukeZhou Feb 18 at 0:18
  • 1
    $\begingroup$ @DukeZhou a hypothesis is ok! Fundamentally that's what I'm looking for. $\endgroup$ – Phylliida Feb 18 at 0:32
  • $\begingroup$ @nbro it seems to me to be more of a situation, but it is also a problem if you phrase it as "what algorithm could cause the AI singularity to happen?". But fundamentally I suppose the answer to your question depends on the mathematical formalization chosen $\endgroup$ – Phylliida Feb 18 at 0:35

Here is one idea. I'll start with a more specific "mathematical singularity", defined as an algorithm that can do the following in N hours or less (for all $N >= 1$):

  1. State equivalent versions (up to notional differences) of all mathematical theorems/conjectures that humans will read and understand in N*20 years after 2018 that can be stated formally in Metamath (this in an arbitrary choice, but Metamath is general enough to include quantum logic and extensions of ZFC so it seems like a decent place to start with. Feel free to instead use Coq, Isabelle, Lean, etc. instead if you prefer), assuming those humans never have access to a "mathematical singularity" capable algorithm and their mathematical community continues living and functioning intellectually in a manner similar in capacity to how it did in 2018
  2. Of those problems, provide correct proofs (these may not be readable, that's ok) of all of those that will be solved by those humans in N*20 years.

This of course does not fully capture all mathematical progress that humans will make in those years: a big component missing is "readable proofs" and concepts that can't be captured in metamath. But it is something that is theoretically formal.

I know that this doesn't include any "continual improvement", what I am referring to here is simply a threshold such that when an algorithm passes it, I think it is sufficently powerful enough to be considered as "intelligent enough" that it has reached close to singularity levels of intelligence. Feel free to adjust the (20 years) constant in your head to match your preferred threshold.

I'm not going to accept this answer because it is lacking "continual improvement", but I brought it up because if we can't figure out how to define it mathematically, perhaps simply having "sufficient criteria" in various domains could be a good start.

Edit: I suppose that the singularity typically involves an assumption of the development of an intelligence that is superior to human society. This implies that it is capable of at least doing the things that our society does, so there is probably a good argument to be made here that "proof accessibility" and "method teachability" are vital to this problem.

I mean, if we think of the current state of the field of calculus, it has gone from an arcane topic only understood by a few field experts, to now being readily accessible and teachable to high school students. While that didn't require proving any new major mathematical theorems, one could argue that much of our technological progress didn't come until advanced mathematical machinery developed (calculus) became accessible to a wide range of people.

I was going to make an argument about how "the difference is that computers can learn quicker: they can read through massive proofs very quickly". But I suppose that depends on the architecture of whatever kind of "thing" is achieving the singularity. I.e., here is a (non-exhaustive) list two possible outcomes:

  • There is only one "mind" that is achieving all of this. In that case, that mind has all the knowledge it needs and it doesn't need to teach anyone to progress further, so this point is sorta irrelevant. However, I can still see an argument for "teachability" if we want to utilize this vast amount of knowledge the AI has gained in human society, if possible.
  • There is a simulated "society" of virtual minds that are interacting with each other, that, together, achieve the mathematical singularity. If a single "mind" in this "society" isn't able to easily use and understand the work done by another mind, then the point of "teachability" is very important to prevent individual minds from having to continually recreate the wheel, so to speak.

Without our biological limitations these digital minds may have very different "teaching" methods, but I think here is the ideal additional requirement for a "mathematical singularity":

  1. These proofs must be (eventually, perhaps not until spending quite a bit of time) accessible to a graduate mathematician, via proving pdf textbooks (or other similar teaching materials) that cover the same material that human mathematical textbooks would have covered after N*20 years in a way that is accessible to the typical graduate mathematician.

However we have now lost some formality in this: textbooks usually contain lots of exposition and analogies that are difficult to formally measure and may not even be relevant for the AI. Here is an alternate option that is not as good, but still close:

  1. The algorithm must present its results in a form that can be used by any other algorithm that also can achieve the "mathematical singularity" to "skip ahead" to N*20 years, and then immediately continue progress from there.

However this criteria has a trivial exploit: an algorithm might as well just provide a 'save state' and a 'program' to run that save state. Conceivably any algorithm that can achieve the mathematical singularity is at least capable of executing code, so providing a 'save state' and 'program' passes this criteria without making it at all accessible (The caveat here is if it uses some sort of model of computation that requires special hardware such as quantum computing or black hole computing to prevent slowdown, but that's besides the point)

I think I prefer this alternative:

  1. These proofs must be similar in length as the (formalized versions of) proofs the human academic community would have made in those 20*N years

"length" is tricky here: it is possible to prove a very difficult theorem very succinctly by simply referencing a very powerful lemma. But here is one example metric:

$$length(Proof) = lengthInSymbols(Proof)+\sum_{symbol \in Proof} \frac{length(symbol)}{numberOfTimesUsedInOtherProofs(symbol)}$$

Where "Other Proofs" is the set of all proofs read and understood by humans in those N*20 years, and "symbols" refers to things such as "Green's Theorem" or "$\in$". Hopefully the idea is apparent here: if something is used frequently in many proofs, it is a "common technique" that isn't vital to that proof, and thus doesn't contribute as much to the "length" of that proof. Finding a potentially more suitable metric here seems like a much more tractable problem then defining the mathematical singularity itself and I suspect this is studied elsewhere more, so I'll leave it at this for now.

  • $\begingroup$ Sorry, but Metamath would lower the quality too much. According to the handbook, it's using the set theory which is outdated. The better way is to think about how to teach modern mathematics to the students so that they will understand it. $\endgroup$ – Manuel Rodriguez Feb 18 at 7:25
  • $\begingroup$ @ManuelRodriguez Could you explain why you think set theory is outdated? Never heard of that before. $\endgroup$ – Oliver Mason Feb 18 at 14:04
  • $\begingroup$ According to the Set theory the world consists of categories. Today, there is something available which is more advanced, it is object oriented programming with Java. As a result, we doesn't need ZFC anymore. $\endgroup$ – Manuel Rodriguez Feb 18 at 14:25
  • $\begingroup$ @ManuelRodriguez I agree Metamath proofs aren't very readable, I edited my answer to clarify that using Metamath specifically isn't important: if you prefer you can substitute that with Coq, Isabelle, Lean, etc. I think you'll find that modern proof systems result in proofs that are much more accessible and readable, and that gap is closing more over time (btw set theory isn't outdated, category theory is basically object oriented programming, but that's besides the point). Anyway, I edited my answer to hopefully better address your point that teachability of mathematics is fundamental here. $\endgroup$ – Phylliida Feb 19 at 22:26
  • $\begingroup$ TIL mathematicians don't need category theory or sets because Java. $\endgroup$ – sfmiller940 Feb 25 at 23:35

The definition of AI singularity can't be done on a purely mathematical basis. Because this would ignore the man-machine interaction. Why is that important? The foundation of artificial Intelligence is located in Cybernetics. Cybernetics is not only a science, but it has to do with how humans and machines are working together. It's correct, that the governor of a steam engine is itself technology, but using the machine in a factory is something else. It's correct, that a chatbot is a computer program, but after starting it, humans will say something to it.

To consider these external factors, it's not enough to use only mathematics in describing AI singularity, but the better way is to focus explicit on mathematics within Cybernetics. This makes clear, that other subjects like philosophy, art and biology are involved too.

  • $\begingroup$ Your argument might be fair, but I figure it is worth at least trying to get as close as possible. This is like saying "gerrymandering can't be defined on a purely mathematical basis", or, from a different angle, "nuclear weapons can't be defined on a purely mathematical basis". While that is technically true (any theoretical model will miss out on some of the practical components, and will miss out on the impact they have in society), there is still value in defining and studying their theoretical approximations because that can bring up useful insight into the problems in the real world. $\endgroup$ – Phylliida Jul 24 at 3:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.