1
$\begingroup$

What would happen when an artificial general intelligence can improve itself over a long time, with limited resources?

The assumption is that it has a large but finite amount of computing power, and can not escape that limit to find resources. Let's assume the limit to be on the order 100 human brains or so - the limits are not important, only that it is limited.

Now, we let it run to self-improve as much as it can with the given resources, until it goes to some stable or periodic state.

What it can do is at least one of the following:

  • converge to some state
  • oscillate between two or more states
  • go into a chaotic state
  • stop doing anything
  • disable itself in some other way than just stop doing anything

There are certainly more complex behaviors possible:

  • become maximally happy, and then periodically redefine what happy means
  • hacking its reward function in some other way

I would expect it to converges to a single state - but that seems naive.

Are there any ideas on how it would end up?

Improve this question
$\endgroup$
3
  • $\begingroup$ To be clear, you're assuming that you have an AGI. Right? So, this question assumes that there's an AGI already that is able to self-improve. Right? You talk about states, but you don't define states, so this is a bit dangerous. I suggest you simplify as much as possible your post and question and try to make it clear what you want to ask and define the terms you are using. This is clearly a philosophical, question, but not only. $\endgroup$
    – nbro
    May 5, 2020 at 20:10
  • $\begingroup$ Also, you talk about "finite resources" but also about "infinite time". Isn't time also a resource? So, you should probably clearly define what you mean by "resource". I am trying to make this question a very good question. I think that it's already interesting, but, hopefully, we can make it more scientifically answerable. $\endgroup$
    – nbro
    May 5, 2020 at 20:11
  • $\begingroup$ @nbro Yes, the term "resource" is too general there. In terms of finite ressources, but infinite time: I do not want to limit how long it takes, I' want to wait until it gets stable or periodic, basically. "Wait until nothing new happens". Yes, that are all good points, thanks! $\endgroup$
    – Volker Siegel
    May 5, 2020 at 20:36

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .