2
$\begingroup$

In the book Artificial Intelligence Engines: A Tutorial Introduction to the Mathematics of Deep Learning, James Stone says

With supervised learning, the response to each input vector is an output vector that receives immediate vector-valued feedback specifying the correct output, and this feedback refers uniquely to the input vector just received; in contrast, each reinforcement learning output vector (action) receives scalar-valued feedback often sometime after the action, and this feedback signal depends on actions taken before and after the current action.

I fail to understand the part formatted in bold. Once we have a set of labeled examples (feature vector and label pairs), where is the "feedback" coming from? Testing and validation results of our calibrated model (say a neural network based one)?

|improve this question
$\endgroup$
2
$\begingroup$

By "immediate vector-valued feedback", they probably mean exactly the label in the "labeled examples" you mentioned.

|improve this answer
$\endgroup$
  • $\begingroup$ Thanks. It does make sense (model dependency of the feedback sounded wrong to me too). Does this mean that, using reinforcement terms, in supervised learning we have the Agent, Action, and a static (single state) Environment, with the reward being immediate and explicit (vector-valued output is correct or not)? Thanks again. $\endgroup$ – ir7 Nov 2 '19 at 19:38
  • $\begingroup$ @ir7 This is a nice way to see it, but also I'd add that the environment is deterministic (whereas in general, in RL the environment is stochastic). Furthermore, using the other way around, you could see RL as a situation where you generate your labeled data dynamically. $\endgroup$ – olinarr Nov 2 '19 at 20:33
1
$\begingroup$

Sorry for the delay. The term "vector-valued feedback" is compared to scalar-valued feedback. The implication (which I should have made explicit) is that, because vector-valued feedback tells the network the correct answer, the changes in weights required to improve performance are reasonably easy to calculate (e.g. using backprop).

In contrast, if a scalar-valued feedback is given (as in reinforcement learning) then the network knows only how bad its previous output was, but not how to change weights in order to improve the output.

A rough analogy would be that vector-valued feedback tells you that you got the wrong answer to a question, and provides the correct answer. In contrast, scalar-valued feedback just tells you 'how wrong' your answer was, but does not tell you how to improve your answer.

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.