2
$\begingroup$

Many experts seem to think that artificial general intelligence, or AGI, (on the level of humans) is possible and likely to emerge in the near-ish future. Some make the further step to say that superintelligence (much above the level of AGI) will appear soon after, through mechanisms like recursive self-improvement from AGI (from a survey).

However, other sources say that such superintelligence is unlikely or impossible (example, example).

What assumptions do those who believe in superintelligence make? The emergence of superintelligence has generally been regarded as something low-probability but possible (e.g. here). However, I can't seem to find an in-depth analysis of what assumptions are made when positing the emergence of superintelligence. What specific assumptions do those who believe in the emergence of superintelligence make that are unlikely, and what have those who believe in the guaranteed emergence of superintelligence gotten wrong?

If the emergence of superintelligence is to be seen as a low-probability event in the future (on par with asteroid strikes, etc.), which seems to be the dominant view and is the most plausible, what assumptions exactly makes it low-probability?

Improve this question
$\endgroup$
1
  • $\begingroup$ Yes, now it clearer! Maybe you could even add my example "e.g. what are the assumptions that people make when they say that recursive self-improvement is possible and will lead to the creation of SI", although that may restrict your question and people could focus on just that, while you're looking for general assumptions. $\endgroup$
    – nbro
    Commented Apr 2, 2020 at 21:19

1 Answer 1

Reset to default
0
$\begingroup$

I am not an expert on the topic, but I will provide some information that could be useful or helpful.

I think that the first and maybe trivial assumption that people make when they say that AGI or SI can emerge is that general intelligence (whatever the definition is) is computable, i.e. there's a Turing machine or, in general, a mathematical model of computation that can simulate an algorithm that possibly represents general intelligence. I don't know how likely the mind of a human (or any other general intelligence) is computable (or to what extent), but there's the so-called computational theory of mind (CTM) that goes into this direction.

An assumption related to the emergence of SI is that recursive self-improvement is physically possible. I also don't know exactly how likely or unlikely recursive self-improvement is, but my limited knowledge of thermodynamics (and physics) suggests that it will not be possible (at least, at the pace or the way people claim or want to suggest). I am not saying that we will not see more progress in the next years (we will), but this doesn't mean that we will be able to create an AGI. People often overestimate themselves and their intelligence.

A third assumption is related to the current achievements in the artificial intelligence field. Many people claim that there has been a lot of progress in recent years and that there's no reason to believe that this will not continue. However, history tells us that the predictions of AI scientists about the future of the AI field were often wrong. There have been at least 2 AI winters because of these wrong predictions and unmet expectations.

Many people also don't exclude the possibility of the emergence of AGI or SI only because they have not yet been proven wrong.

Another assumption is that there won't be any catastrophe (like COVID-19 can potentially be) that will significantly slow down the general scientific progress.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for your answer! I have some questions: (1) why does it matter if intelligence is computable? Why can't AGI be based on a neural network fed massive amounts of data, or a very advanced semantic reasoner? (2) I agree with your other points, but I'm wondering if you know of any sources that discuss the speed at which recursive self-improvement could happen? For example, it's conceivable that improvement takes longer and longer each iteration (even if the machine is smarter, it has too much computation to do, etc.). Any sources on this? $\endgroup$
    – user35673
    Commented Apr 3, 2020 at 17:29
  • $\begingroup$ @user35673 (1) That's the point. If you have a neural network that can be used to represent GI, then GI is computable. However, the converse may not be true, i.e. if GI is computable, it doesn't mean that there's a NN that can compute/represent it. Anyway, how do we know that GI is computable? And what is the definition of GI? These questions need to be answered before attempting to implement GI. There are some definitions of GI. For example, have a look at AIXI (which is actually incomputable, although there are computable approximations!). $\endgroup$
    – nbro
    Commented Apr 3, 2020 at 18:00
  • $\begingroup$ @user35673 (2) Honestly, no. Maybe have a look at Bostrom's book "Superintelligence: Paths, Dangers, Strategies". I had read like the first 100 pages of that book, which goes into this direction, but I don't guarantee that you will find exactly what you're looking for there. Maybe have a look at this people.idsia.ch/~juergen/history.html. $\endgroup$
    – nbro
    Commented Apr 3, 2020 at 18:03
  • $\begingroup$ I see, thank you. So would the following be a fair summary of the current state?: AGI is possible, but we have no clue on when or whether it will exist (because of problems like the computability of human-level intelligence). Even if AGI does emerge, it is unlikely it will achieve superintelligence because of the reasons outlined here. And, finally, if superintelligence is achieved, we do not know if it matters too much (e.g. it takes too much computing power, or is simply not that useful). $\endgroup$
    – user35673
    Commented Apr 3, 2020 at 19:49
  • $\begingroup$ If this is an accurate summary, I can certainly see why people think SI is extremely unlikely. To expand on my last point ("simply not that useful"), it is conceivable to me how even the smartest agent could not solve global warming by itself. Or, even, that it is given the instruction to by humans. Presumably, if humans develop this kind of SI, they would be unbelievably careful with its usage. $\endgroup$
    – user35673
    Commented Apr 3, 2020 at 19:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .