How is the reward in reinforcement learning different from the label in supervised learning problems?

Question

How is the notion of immediate reward used in the reinforcement learning different from the notion of a label we find in the supervised learning problems?

Answer 1

Reward in reinforcement learning (RL) is entirely different from a supervised learning (SL) label, but can be related to it indirectly.

In a RL control setting, you can imagine that you had a data oracle that gave you SL training example and label pairs $x_i, y_i$ where $x_i$ represents a state and $y_i$ represents the correct action to take in that state in order to maximise the expected return. For simplicity I will use $G_t = \sum_{k=1}^{\infty} \gamma^k R_{t+k+1}$ for return here (where $G_t$ and $R_t$ are random variables), there are other definitions, but the argument that follows doesn't change much for them.

You can use the oracle to reduce the RL training process to SL, creating a policy function $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ learned from a dataset that the oracle output. This clearly relates SL with RL, but how do $x_i, y_i$ from SL relate to $s_t, a_t$ from RL in terms of reward values?

The states can relate directly (as input):

$$x_i \equiv s_t$$

The action from the policy function is more indirect, if you want to see how reward is involved:

$$y_i \equiv \pi^*(s_t) = \text{argmax}_a \mathbb{E}_{A \sim \pi^*}[\sum_{k=1}^{\infty} \gamma^k R_{t+k+1} | S_t=s_t, A_t=a]$$

Note the oracle is represented by the optimal policy function $\pi^*(s_t)$, and the expectation is conditional both on the start conditions of state and action plus following the optimal policy from then on (which is what $A \sim \pi^*$ is representing).

In practice the optimal policy function is unknown when starting RL, so the learning process cannot be reduced to a SL problem. However, you can get close in some circumstances by creating a dataset of action choices made by an expert at the problem. In that case a similar relationship applies - the label (of which action to take) and immediate reward are different things but can be related by noting that the expert behaviour is close to the $\text{argmax}$ over actions of expected sums of future reward.

Another way to view the difference:

• In SL, the signal from the label is an instruction - "associate these two values". Data is supplied to the learning process by some other independent process, and can be learned from directly

• In RL, the signal from the reward is a consequence - "this is the value, in context, of what you just did", and needs to be learned from indirectly. Data is not supplied separately from the learning process, but must be actively collected by it - deciding which state, action pairs to learn from is part of the agent's learning task