2
$\begingroup$

I've learned that you can use two perceptrons to ultimately create a classifier for non-linearly separable data. I'm trying to understand how / if these two perceptrons converge to two different decision boundaries though.

enter image description here

Source: https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html

I don't understand how the second perceptron creates a different decision boundary when it has the same input as the first perceptron? I know the weights can be initialized differently but does this second perceptron classify something else? Shouldn't the decision boundaries be converge to be the the same ultimately after training?

enter image description here

Source: https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html

$\endgroup$
2
$\begingroup$

I think your confusion comes from the fact that you are calling those two hidden nodes "perceptrons". You shouldn't call the hidden nodes in your network perceptrons. You should call them "nodes" or "neurons", although the term "multilayer perceptron" comes exactly from that.

You should think of a perceptron as something like

enter image description here

A perceptron simply computes a linear combination of the inputs using the corresponding weights. A single perceptron can (only) linearly separate a set of points.

However, if you have two "stacked perceptrons" (like the hidden nodes in your network), then, provided that both "perceptrons" (or hidden nodes) only compute a linear combination of the inputs, the output of each hidden node can be interpreted as line (like the violet and green lines in your picture).

I don't understand how the second perceptron creates a different decision boundary when it has the same input as the first perceptron?

This is because the two perceptrons will eventually have different weights. You can see from your diagram that $p_0$ has weights $w_{0, 0}$ and $w_{0, 1}$, whereas perceptron $p_1$ has weights $w_{1, 1}$ and $w_{1, 0}$. Note that weights are learned, during training, depending on your loss function (at the output node $p_2$) and the labeled dataset (that you use to train that simple neural network).

I know the weights can be initialized differently but does this second perceptron classify something else?

If you consider the two hidden nodes (in your network) separately, then (as I said above) the output of these two hidden nodes can be interpreted as a line, and these lines might be different (as illustrated in your picture), because these two hidden nodes will eventually learn something different about the input. However, you should also think of the output of two hidden nodes as contributing to the input of the output node. So, in a certain way, you're combining the decisions of the two hidden nodes (or, as you call them, "perceptrons").

Note that you will need to introduce "non-linearity" in those hidden perceptrons in order to be able to non-linearly separate a set of points.

Shouldn't the decision boundaries both converge to the same ultimately after training?

No, not necessarily. The decision boundary found by the perceptron learning algorithm depends on the initialization of the weights and the order that the inputs are presented. See chapter 4 (specifically, pages 192-196) of Pattern Recognition and Machine Learning by C. Bishop. Two perceptrons would be guaranteed to converge to the same solution if the loss function was convex (i.e. there's only one optimum).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.