Here are some problems that my ape mind came up with.
1.Smart != all knowing
The AI self improvement explosion smakes it smarter and smarter. Being smarter doesn't mean knowing more facts. I suppose this is quite a picky argument, but I think worth thinking about.
A very smart doctor who doesn't know your history may still make a worse choice than a less intelligent one with better data.
2. Is it all for humans? Is it for all humans?
ASI which reaches a higher level could not be interested in our wellbeing.
A controlled ASI could still work for the benefit of the few only, if these few decide wrong goals we could go backwards.
3 Harsh ASI
A scientific mind in not necessarily full of sympathy or empathy.
4. Being smart and not being clever
Great minds still make mistakes:
* in setting their goals, and
* in executing the plan to achieve it.
Geat intellect doesn' guarantee lack of shorosightedness or lack of blind spots.
If there are bounds of existence (speed of light type limits) then the AI will be bound by these as well. This may mean that there are things that even ASI won't 'get'.
Also, as our mind may have limits based on its structure the next AI may have limits as well - and even if it improves uppn may hit limits that it cannot find solutions to because it's 'too stupid'.
6. We won't get it
ASI's understanding of certain aspects of the world may not be communicative to most humans. We just won't get it (even if we're capable of understanding everything, it doesn't mean we will understand it).
7. How to use it?
We may destroy ourselves, and the AI with the tech it helps us build. It doesn't need to be bombs. It can be geoengineering or wonder drugs.
This is especially intense when the ASI is already powerful but not strong enough to forsee negative consequences (or we'll just ignore it anyway).