Is reinforcement learning only about determining the value function?

Question

I started reading some reinforcement learning literature, and it seems to me that all approaches to solving reinforcement learning problems are about finding the value function (state-value function or action-state value function).

Are there any algorithms or methods that do not try to calculate the value function but try to solve a reinforcement learning problem differently?

My question arose because I was not convinced that there is no better approach than finding the value functions. I am aware that given the value function we can define an optimal policy, but are there not other ways to find such an optimal policy?

Also, is the reason why I don't encounter any non value-based methods that these are just less successful?

Answer 1

There are many algorithms that are not based on finding a value function. The most common ones are policy gradients. These methods attempt to map states to actions through a neural network. They learn the optimal policy directly, not through a value function.

The important part of the image is when Model-Free RL splits into Policy Optimization (which includes policy gradients) and Q-Learning. Later you can see the two sections coming back together in algorithms that are a mix of both techniques. Even the bottom three methods in policy optimization involve some form of learning a value function. The best and most advanced algorithms use value function learning and policy optimization. The value function is only for training. Then when the agent is tested, it only uses the policy.

The most likely reason you have only heard of value function methods is because policy gradients are more complicated. There are many algorithms more advanced than ones that only use value functions and policy gradients can learn to operate in continuous actions spaces (an action can be between -1 and 1, like when moving a robot arm) while value functions can only operate with discrete action spaces (move 1 right or 1 left).

Summary: Yes, there are other methods that learn the optimal policy without a value function. The best algorithms use both types of reinforcement learning.

The SpinningUp website has a lot of information about reinforcement learning algorithms and implementations. You can learn more about direct policy optimization there. That is also where I got the image from.

This answer is specific to the most common types of Model-Free RL. There are other algorithms related to the RL problem that do not learn value functions, like inverse reinforcement learning and imitation reinforcement learning.