The following text is from Hal Daumé III's "A Course in Machine Learning" online text book (Page-41).

enter image description here

I understand that $D$ is the size of the input vector $x$.

  1. What is $y$? Why is it introduced in the algorithm? How/where/when is the initial value of $y$ given?

  2. What is the rationale of testing $ya \leq 0$ for updating weights?


Y is the desired output of the perceptron (often referred to as target) , for the given set of input vectors.

Rationale behind Y.a<=0 :

Prerequisite knowledge :

  • A=A-B : Moves vector A away from direction of vector B

  • A=A+B : Moves A in the direction of B

  • A (.) B >0 ; A vector is directed acutely (<90 deg.) towards B vector

  • A (.) B <0 ; A vector is directed away from (>90 deg.) from B vector [(.) denotes dot (scalar) product and bold letters indicate vectors]

  • W is augmented vector (includes threshold as another weight along with normal input weights)

  • X is augmented input (includes -1 as extra input (corresponding to threshold) along with other normal inputs

  • a(activation) = W (.) X

  • a >=0 ; Perceptron output 1

  • a<0 ; Perceptron output -1 (Not zero as implicit in the given algorithm I think)

Now Rationale :


This means Either of the following :

  • Y=-1 and a>0 ; in this case the target output is -1 but as a>0 so the Perceptron outputs 1. So we must move the weight vector away from this set of input vector, so that angle between them increases and the dot product (a) becomes < 0 so that we can get the target output. Hence : W=W-X

Or ,W=W + (-1)*X Or, W=W+YX

  • Y=1 and a<0

This means the target output is 1 but as the activation is <0 so Perceptron is outputting -1. So we must move the weight vector close to this set of input vector so that the activation can become >0 (angle decreases) and the Perceptron can output the desired output. So : W = W+X Or W=W + YX

Again , Y.a=0 is a boundary case. By now I think you can understand the rationale behind Y.a=0. If any doubt , comment to this answer , I will explain it.

Sorry for so much long answer though. :) :)

| improve this answer | |
  • $\begingroup$ Is that $W(.)A = = W^T.A $? $\endgroup$ – user3642 Dec 26 '16 at 7:47
  • $\begingroup$ What are A and T in the above comment? $\endgroup$ – Programmer Dec 26 '16 at 8:15
  • $\begingroup$ Sorry for that, W(.)X == W^T.X. I.e. dot product of transpose of W with X. $\endgroup$ – user3642 Dec 26 '16 at 8:51
  • $\begingroup$ No they won't be same until you perform transpose on both the vectors. [ Although I assume that by transpose you mean reversing the components of the vectors. I.e , xi+yj+zk -> zi+yj+xk ] as transpose is inherent concept with matrix not vectors. $\endgroup$ – Programmer Dec 26 '16 at 9:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy