How is the notion of immediate reward used in the reinforcement learning different from the notion of a label we find in the supervised learning problems?

  • 1
    $\begingroup$ Does this answer your question? Can supervised learning be recast as reinforcement learning problem? This is not an exact duplicate, but your question is very related to the other one. If you think the answer and question in the other link are useful, feel free to upvote both of them. $\endgroup$ – nbro Jul 7 at 15:17
  • 1
    $\begingroup$ @nbro Indeed both the question and the answer that you gave is very interesting (sadly I don't have enough reputation to upvote them right now). But my question is slightly different. I'm not seeking to convert one type of problem to another. My intention is to know how much do they resemble and contrast each other. $\endgroup$ – Saptam Jul 7 at 15:40

Reward in reinforcement learning (RL) is entirely different from a supervised learning (SL) label, but can be related to it indirectly.

In a RL control setting, you can imagine that you had a data oracle that gave you SL training example and label pairs $x_i, y_i$ where $x_i$ represents a state and $y_i$ represents the correct action to take in that state in order to maximise the expected return. For simplicity I will use $G_t = \sum_{k=1}^{\infty} \gamma^k R_{t+k+1}$ for return here (where $G_t$ and $R_t$ are random variables), there are other definitions, but the argument that follows doesn't change much for them.

You can use the oracle to reduce the RL training process to SL, creating a policy function $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ learned from a dataset that the oracle output. This clearly relates SL with RL, but how do $x_i, y_i$ from SL relate to $s_t, a_t$ from RL in terms of reward values?

The states can relate directly (as input):

$$x_i \equiv s_t$$

The action from the policy function is more indirect, if you want to see how reward is involved:

$$y_i \equiv \pi^*(s_t) = \text{argmax}_a \mathbb{E}_{A \sim \pi^*}[\sum_{k=1}^{\infty} \gamma^k R_{t+k+1} | S_t=s_t, A_t=a]$$

Note the oracle is represented by the optimal policy function $\pi^*(s_t)$, and the expectation is conditional both on the start conditions of state and action plus following the optimal policy from then on (which is what $A \sim \pi^*$ is representing).

In practice the optimal policy function is unknown when starting RL, so the learning process cannot be reduced to a SL problem. However, you can get close in some circumstances by creating a dataset of action choices made by an expert at the problem. In that case a similar relationship applies - the label (of which action to take) and immediate reward are different things but can be related by noting that the expert behaviour is close to the $\text{argmax}$ over actions of expected sums of future reward.

Another way to view the difference:

  • In SL, the signal from the label is an instruction - "associate these two values". Data is supplied to the learning process by some other independent process, and can be learned from directly

  • In RL, the signal from the reward is a consequence - "this is the value, in context, of what you just did", and needs to be learned from indirectly. Data is not supplied separately from the learning process, but must be actively collected by it - deciding which state, action pairs to learn from is part of the agent's learning task

| improve this answer | |
  • $\begingroup$ Excellent answer. Exactly what I was looking for. Indeed, this indirect dependency (which I was missing) is what differentiates them. On a side note, from what I've read the reward distribution can be noisy. Is it possible in supervised learning that the labels themselves are less reliable? We usually take it for granted that the labels are produced by some almighty oracle. $\endgroup$ – Saptam Jul 8 at 9:45
  • $\begingroup$ @Saptam: Yes, supervised learning works with noisy labels, and that is a common assumption. Most SL methods will do the correct thing of learning an expected value based on statistics of the labels in the dataset. $\endgroup$ – Neil Slater Jul 8 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.