Classification with different approaches

Question

A few days ago I asked the question, if a NN with linear activation function can produce a function concatenated of linear functions, what actually is impossible (Can a NN with linear activation functions produce a connection of linear functions?).

Now I have here some classification examples, but I really cannot perfectly decide, which one is based on which approach:

1 -> C The perceptron does not look for the maximum separation margin.

2 -> E Neural network with linear activation function

3 -> A Linear SVM, because of the maximum separation margin.

4 -> B Because of the hyperbolic shape of the hyperplane.

5 -> D? Logisitc regression? I tought it can only linear separate?

6 -> F I guess the NN with tanh activation function, because of the no very smooth shape, which comes from the too small hidden layer size.

I actually don't get how the logistic regression classifier should be able to produce a hyperplane like in 5? What did I classify wrong here?

Answer 1

As far as I understand your question, you are trying to match the learning algorithm with their results.

Firstly, I don't think the answer mapping you have provided is correct.

First, we see the options and find out which one linearly seperates the data: We have 3 candidates for the job:

A. Linear SVM

C. Perceptron

D. Logistic Regression

In the datasets only 1, 2 and 3 are linearly seperated. Since, we know linear SVM will search for decision boundary with maximum separation between 2 linearly separable data sets hence A goes with 3. Pereceptron is basically the linear version of logistic regression. Now, LR is an exponential algorithm hence the error does not appear as large as it should. Say that for a particular input vector x the y = 1.Say the hypothesis (x * transpose(theta)) your algorithm generated is 5. For perceptron error will be 5 - 1 = 4 whereas for LR it will be 1 / (1 + e^(-5)) - 1 = 0.9932. If hypothesis generates 6 error will be 5 for perceptron and about 0.9974 for LR. So we see even for very great increase in errors the cost for LR will not increase significantly (inherent property of exponentials) whereas it increases linearly for perceptron.

Hence, 1 goes with C since it tries to minimize the error, whereas 2 will go with D since even though there is a greater cost it gets scaled due to exponential function.

And now I think 5 will go with F since the curve has a smoothness associated with it, whereas due to Re-Lu's the curve in 6 will go to E (due to an associated sharpness) and finally B goes to 4 as it has a kernel of order 2 and x^2/a^2 + y^2/b^2 +xy = c is the equation of a hyperbola of order 2.

Hence final mapping:

• A - 3
• B - 4
• C - 1
• D - 2
• E - 6
• F - 5

I may have missed few other mathematical subtleties, and I would be grateful if someone points them out.