The short answer is that no one knows - here is just my take on it.
Note here that this all sounds SciFi as hell. This didn't occur yet to any degree but is in the domain of the physical possible.
I will divide this into Hardware, Software and Economics. Economics is just a replacement for the name of a concept which I am unaware of - that is the interaction between A(G)I's, companies and companies of AGI's.
Note that the growth of GPU's outpaces moores law. Note also that the growth of the computational power throws at ML outpaces moores law, too AI and Compute
Software improvements are certainly possible.
A lot of software improvements can be found in Levels of the self-improvement of the AI.
The software improvements of the core A(G)I mechanisms, algorithms and datastructures depend on
a) The intelligence and optimization power of the used A(G)I(s).
Incapable A(G)I's may either fail completely to improve the algorithms and datastructures and intelligence or they cap out at a certain (sub)human level. See Analysis of Types of Self - Improving Software Timeline and analysis of existing attempts of recursive self improving (RSI) software systems for a detailed analysis and a analysis of most publically known attempts.
b) The already done optimizations of the used software (of the A(G)I(s)) and the format in which the programs are encoded.
All current ML frameworks make certain assumptions and tradeoffs between (software) optimizations and ease of use for human programmers. The efficiency of lets say Tensorflow can be greatly improved for most special cases.
Incapable A(G)I's may require the description of the algorithms with DSL's because they fail to reason about contemporary programming languages (C, C++, Java, etc) and frameworks (Tensorflow, etc.). Further they may require speciallized training and/or software to develop software.
c) The mental architecture of the A(G)I's
Some architectures might be easier to improve by the A(G)I's than others. One extreme is where the AGI was built by an specialized AI - here everything is "in the mind" of the AGI. One other extreme is a architecture which is based on a biological brain. This might be extremely hard or impossible to improve uppon because natural evolution had done all optimizations it could do in 100's of millions of years.
Specialized Hardware is faster than Software. Examples for this are uses of FPGA's, GPU's and custom ASICs.
Speedup of FPGA code depends on the used FPGA and the algorithm and datastructures as well as the implementation itself. Typically in the range on ~100x - 10000x (?) vs CPU's.
Speedups of GPU code depends on a lot of factors - but are usually between 5x to serveral 100 x. This all depends on the optimizations done on the CPU software which are used for comparision.
A example for custom ASIC's are "Tensor Processing Units". The speedup of TPU's today is ~2.8x against GPU's - depending on the workload Benchmarking Google’s new TPUv2. Nothe here that custom ASIC's may be more energy efficient than CPU's, GPU's or FPGAs.
Lets assume that a sufficiently developed AGI is used by entities which ressemble companies of today (2019).
We can put entities on a graph where the X axis is the amount of work done (by some computable or not computable metric) and the Y axis is the ration between human labor and work done by AGI's.
Purely human driven entities are essentially like companies of today - they may use weak AI for some tasks (like everyone is doing today - weak AI is everywhere).
There is a interesting regime where a lot of work is done by humans and AGI's. AGI's might get tasked with all sorts of tasks. Interesting tasks worth to mention here include company leadership and decision making for the company.
This comes with it's own social problems inside a mixed company - who would like to work for an AGI where the AGI is the boss and decides on the fate of the company.
A mixed version of the "AGI leadership" idea are humans which use AGI's as an oracle to make decisions.
There are lots of possibilities of only this interaction.
The other extreme regime is where almost all or all work is done by AGI's. Human operators may or may not exist and excert power over the AGI's.
Note here that many AGI's may act like a single AGI. One other interesting effect is that AGI's may decide to split off and form their own "company" - or they might decide to fuse.
This occurs today with contemporary companies but at a slow pace. AGI's may do this in a very short time (minutes, seconds, etc).
Then there will be basically three different company like entities - purely human driven ones, ones where there is a mix of AGI's and humans and the third category is made up of almost only AGI's.