The short answer is that it hasn't received nearly the attention it should have despite the cited paper having over 600 citations.
According to this Singularity Summit talk by Shane Legg, his PhD thesis on the wide ranging uses of the word "intelligence" managed to identify 2 qualitative dimensions of those uses:
- Human vs Ideal
- Internal vs External
Of the 4 points in the space of these 2 dimensions, the sense in which his paper with Hutter uses the word "intelligence" is (Ideal, External). This is distinguished from the Turing Test which is for (Human, External).
The vast majority of machine learning targets (Ideal, External) intelligence rather than modeling human intelligence.
The formal, "top-down" definition of (Ideal, External) provided by Hutter is called AIXI, which is the unification of ideal science (Algorithmic Information Theory aka AIT) with ideal technology (Sequential Decision Theory aka SDT).
Unlike other attempts to define (Ideal, External) intelligence, it has only 2 open parameters:
- Choice of Universal Turing Machine running the universe in which the agent perceives as well as the agent's simulation of that universe to make predictions.
- Choice of utility function providing the value system of the agent so it can decide which actions yield the most valuable consequences.
In the vernacular, AIT is about what "is" and SDT is about what "ought" to be done about what "is".
For just one example of why AIXI has not received nearly the attention it should have:
The ethics of AI are riddled with the conflation of "is" with "ought". Under AIXI, these can be, at the top level of analysis, factored out using the Algorithmic Information Criterion for model selection, which is conceptually quite simple:
Given the union of all datasets used to train "large" AI models, choose the model that yields the smallest executable archive of that data.
The model so-selected is the best available model of what "is" the case -- including models of bias in the (possibly latent) identities that are the sources of data. If we are to be fair about what is and is not "bias" here's how to proceed:
If someone has a reason to call some data "biased" then they should be challenged to provide the data that supports their perception of what is and is not "biased". Then, simply include that data in the total dataset. If they, themselves, are biased and select biased data because of their bias, let others include the data that shows that bias to be, in fact, bias. Compress relentlessly until the data that is most consilient with the rest of the universe is better compressed, leaving the bias exposed as "noise" associated with the responsible identities.
This has applications in dealing more effectively not only with social media censorship, but by factoring out the "is" from the ethics, it permits people to recognize when they simply differ in their utility functions or "value systems" and get on with the hard problem of dealing with quasi-religious differences and thereby reduce the likelihood of conflicts that can be exceedingly destructive.
Nor is this merely pie-in-the-sky as Hutter has demonstrated with The Hutter Prize for Lossless Compression of Human Knowledge. The Hutter Prize, if funded at the level deserving of the problem of ethics in AI, would be several orders of magnitude larger.