My background is in political economy and game theory. I am interested in the discussion on AI risk and alignment, but I have so far failed to find work on this that seriously engages with classic axiomatic rational choice theory (RCT).

Some claims are often made that, to me, contradict the basic tenets of RCT. For example, it is often suggested that a sufficiently advanced AGI might come up with goals of its own that put at risk humanity. However, according to RCT, agents can only choose actions, whereas their preferences (goals) are primitives of the model. It is nonsensical to think of an agent as choosing his goals, except in some well-defined dynamic model (say, Becker's model of rational addiction) where the future preferences result from actions that can be derived, by equilibrium analysis, from primitive preferences.

Another claim that perplexes me is that the actions of a sufficiently intelligent AI may be impossible to predict. However, to me the more intelligent an agent, the more its actions should approximate those of an expected utility maximizer.

I wonder if serious people are working on this from a different axiomatic system. What are some good sources (at any level of technicality) that I should read? Better if books.

  • $\begingroup$ Human Compatible by Stuart Russell? $\endgroup$ Nov 28, 2023 at 18:21
  • $\begingroup$ I'm looking for something a bit more rigorous: something structured as Definition-Lemma-Theorem, in the tradition of rational choice / game theory. $\endgroup$ Nov 28, 2023 at 20:44


You must log in to answer this question.

Browse other questions tagged .