What are reward networks in reinforcement learning?

Question

I am reading the following article given over here -

The goal of both inverse reinforcement learning (IRL) algorithms (e.g. AIRL, GAIL) and preference comparison is to discover a reward function. In imitation learning, these discovered rewards are parameterized by reward networks.

Does this mean that the output of imitation learning is a reward value? What are the inputs then? The state and action?

I am currently referring to Levine's lecture notes. At slide 5, it seems like the output of imitation learning is a policy and not a reward.

I'd appreciate an answer that refers to papers while explaining these concepts.

Answer 1

In RL, reward networks (also called reward models), say $r_\theta$, have the sole role of learning (approximating) the reward function, defined as $r(s,a)$ or $r(s, a,s')$, in which the latter formulation is more general. A reward function, $r$, takes the current state $s$, selected action $a$ (by a policy, expert trajectories, or behavioral policy), and optionally the next state $s'$, to output the immediate reward, $r_t$, of timestep $t$. Reward networks should approximate $r_t$, and are employed in inverse reinforcement learning (IRL) and sometimes also in model-based RL.

Moreover, you should not confuse IRL with imitation learning (IL), as IL is more general because it also includes supervised learning methods such as behavioral cloning. In IRL, you want to learn a reward function first that is usually parameterized by a NN (therefore you have a reward network), and then use standard model-free algorithms from the learned reward to learn the policy. This is the general setting, but GAIL for example can learn the policy directly.

Note also that when you do IRL or IL you don't have access to the environment, and therefore you don't know the true reward function, but you have only access to a finite number of samples (called demonstrations) which can be from either an expert, exploratory, or sub-optimal policy.

You can refer to this survey for a comprehensive introduction about IRL, its foundations, flavors, and applications.

Answer 2

One representative paper for imitation learning is Ho & Ermon's "Generative Adversarial Imitation Learning" (2016). The goal in imitation learning is to learn a policy that can mimic the expert's behavior.

Consider learning a policy from example expert behavior, without interaction with the expert or access to reinforcement signal. One approach is to recover the expert’s cost function with inverse reinforcement learning, then extract a policy from that cost function with reinforcement learning. This approach is indirect and can be slow. We propose a new general framework for directly extracting a policy from data, as if it were obtained by reinforcement learning following inverse reinforcement learning.

Behavioral cloning, while appealingly simple, only tends to succeed with large amounts of data, due to compounding error caused by covariate shift [23, 24]. Inverse reinforcement learning (IRL), on the other hand, learns a cost function that prioritizes entire trajectories over others, so compounding error, a problem for methods that fit single-timestep decisions, is not an issue... Unfortunately, many IRL algorithms are extremely expensive to run, requiring reinforcement learning in an inner loop.

We desire an algorithm that tells us explicitly how to act by directly learning a policy. To develop such an algorithm, we begin in Section 3, where we characterize the policy given by running reinforcement learning on a cost function learned by maximum causal entropy IRL [31, 32]... The discriminator network can be interpreted as a local cost function providing learning signal to the policy—specifically, taking a policy step that decreases expected cost with respect to the cost function.

Therefore GAIL is mainly imitation learning which learns a policy as its ultimate goal, yet it also has a discriminator network similar to GANs that acts as a local cost function and aims to distinguish expert trajectories against learned policy's trajectories, establishing a more rapid framework for simultaneously learning policy and reward from data due to its adversarial nature and bypassing an additional independent IRL step.

Answer 3

I think the reason why some refer to learning a reward policy and others learning an actor policy is that both are possible.

For example, GAIL: A discriminator (the reward policy) learns to tell apart expert trajectories and the actor's trajectories, and the actor tries to fool the discriminator by learning to imitate expert trajectories.

In this way, both a reward policy and actor policy are learnt simultaneously.