What would happen when an artificial general intelligence can improve itself over a long time, with limited resources?
The assumption is that it has a large but finite amount of computing power, and can not escape that limit to find resources. Let's assume the limit to be on the order 100 human brains or so - the limits are not important, only that it is limited.
Now, we let it run to self-improve as much as it can with the given resources, until it goes to some stable or periodic state.
What it can do is at least one of the following:
- converge to some state
- oscillate between two or more states
- go into a chaotic state
- stop doing anything
- disable itself in some other way than just stop doing anything
There are certainly more complex behaviors possible:
- become maximally happy, and then periodically redefine what happy means
- hacking its reward function in some other way
I would expect it to converges to a single state - but that seems naive.
Are there any ideas on how it would end up?