Why is reinforcement learning not the answer to AGI?

Question

I previously asked a question about How can an AI freely make decisions?. I got a great answer about how current algorithms lack agency.

The first thing I thought of was reinforcement learning, since the entire concept is oriented around an agent getting rewarded for performing a correct action in an environment. It seems to me that reinforcement learning is the path to AGI.

I'm also thinking: what if an agent was proactive instead of reactive? That would seem like a logical first step towards AGI. What if an agent could figure out what questions to ask based on their environment? For example, it experiences an apple falling from a tree and asks "What made the Apple fall?". But it's similar to us not knowing what questions to ask about say the universe.

Answer 1

Some AI researchers do think RL is a path to AGI, and your intuition about how an agent would need to be proactive in selecting actions to learn about is exactly the area these researchers are now focused on.

Much of the work in this area is focused on the idea of curiosity, and since 2014 this idea has gained a lot of traction in the research community.

So, maybe RL can lead to AGI. We don't know for sure yet.

However, many of the classic arguments against AGI aren't addressed by the RL approach. For instance, if like Searle, you think computers just don't have the right kind of hardware to do thinking, then running an RL algorithm on that hardware isn't going to yield AGI, just ever increasingly robust narrow AI. Ultimately Searle's arguments get into issues of metaphysics, so it isn't clear that there exists any argument that would convince someone like Searle that a particular computer-based technique is AGI-capable.

There are also other arguments. For example, the cognitivist school of thought thinks that statistical learning approaches to AI, and, in particular, the black-box approaches of statistically-driven RL, are unlikely to lead to general intelligence because they do not engage in the kind of systematic reasoning process that proponents of cognitivism assume is necessary for general intelligence. Some more extreme proponents of this school might say that a logical planning algorithm like STRIPS is innately more intelligent than any approach based on deep learning, because it involves sound logical deduction rather than mere statistical calculation. In particular, STRIPS can correctly generalize to any new domain, as long as it is fed the correct sense data, while an RL approach will need to learn how to act there.

So, while there are definitely reasons to be optimistic about RL as a direction for achieving AGI, it's definitely not yet settled.

Answer 2

A relatively recent but interesting paper that discusses this topic in more detail is Reward is enough (Artificial Intelligence, 2021) by David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton (so by some of the godfathers of RL, who are all at DeepMind).

Their reward-is-enough hypothesis (RIEH) (page 4) is

Hypothesis (Reward-is-Enough). Intelligence, and its associated abilities, can be understood as subserving the maximisation of reward by an agent acting in its environment.

This hypothesis is slightly different from the reward hypothesis (RH), which states that all goals can be represented by rewards and the achievement of those goals can be viewed or formulated as the maximization of rewards, because the RIEH also states that the abilities needed to achieve the main goal in the environment arise from the maximization of the reward, so the RIEH is a stronger hypothesis than the RH.

The authors give examples to explain the RIEH (emphasis mine).

Sophisticated abilities may arise from the maximisation of simple rewards in complex environments. For example, the minimisation of hunger in a squirrel’s natural environment demands a skilful ability to manipulate nuts that arises from the interplay between (among other factors) the squirrel’s musculoskeletal dynamics; objects such as leaves, branches, or soil that the squirrel or nut may be resting upon, connected to, or obstructed by; variations in the size and shape of nuts; environmental factors such as wind, rain, or snow; and changes due to ageing, disease or injury. Similarly, the pursuit of cleanliness in a kitchen robot demands a sophisticated ability to perceive utensils in an enormous range of states that includes clutter, occlusion, glare, encrustation, damage, and so on.

They also try to argue why language, perceptron, social intelligence and general intelligence could all arise from the maximization of a single reward signal (e.g. survival).

Moreover, they also say that similar sophisticated abilities associated with intelligence could arise from the maximization of different reward signals, i.e. the emergence of these abilities is robust to the choice of reward objective.

Additionally, they also talk about prior knowledge and learning, but, in my view, they should have emphasized/noted that, for example, perception, without the suitable sensors (inductive bias) cannot emerge: this is not a limitation of the RIEH, as it says nothing about how these abilities actually arise, or the nature of the agent needed for them to arise, or which specific reward signal should be maximized.

In the end, they also conjecture that RL is the main framework that could be used to find out whether these conjectures/speculations are true or not.

They do not go into philosophical arguments, such as the Chinese-Room argument or the problem consciousness: their argument to address these issues would probably be that any ability (even consciousness, if it's an ability) required to achieve the ultimate goal would arise in the process of maximization of the reward.