What are the differences between softmax regression and logistic regression (other than when the number of classes is 2)?

Question

I read about softmax from this article. Apparently, these 2 are similar, except that the probability of all classes in softmax adds to 1. According to their last paragraph for number of classes = 2, softmax reduces to LR. What I want to know is other than the number of classes is 2, what are the essential differences between LR and softmax. Like in terms of:

• Performance.
• Computational Requirements.
• Ease of calculation of derivatives.
• Ease of visualization.
• Number of minima in the convex cost function, etc.

Other differences are also welcome!

I am asking for relative comparisons only, so that at the time of implementation I have no difficulty in selecting which method of implementation to use.

Answer 1

As written, SoftMax is a generalization of Logistic Regression.

Hence:

1. Performance: If the model has more than 2 classes then you can't compare. Given K = 2 they are the same.

2. Computation Requirements: Please explain as the computational requirements require the data, enough memory to hold it and enough time to let run.

3. Ease of Calculation of Derivatives: The cost function is summation hence once you do it for one element you do it fol all.

4. Ease of Visualization: Well, it is easy to visualize the Confusion Matrix even for K = 10 classes. So no issue here.

5. Cost Function: The cost function is convex. Yet not Strictly Convex hence there infinite number of minima.

Answer 2

I don't think that it's useful to differentiate logistic regression and softmax based on your terms. This is because you don't choose one or the other based on performance/computational requirements/ease of calculation of derivatives/...

The fact is that you use one or the other based on which is your problem.

If you need to recognize cat pictures vs. non-cat pictures you will use logistic regression (even with a very complex NN the last step will be always a logistic regression). Of course, you could use softmax but the outputs will be redundant, i.e. one output will always be one minus the other.

If you need to recognize cat pictures vs. dog pictures vs. other pictures you will use softmax. Note that, in order to use softmax, you need to have only mutually exclusive classes. Mutually exclusive classes mean that an example cannot belong to multiple classes. What if a picture represents both a dog and a cat? In this case, it should be marked as other picture. If you want to avoid these you could use one more class to denote pictures with both cats and dogs.

However, if you want to recognize cats, dogs, birds, fishes, boats, houses, etc. the number of mixed classes that you need to include will grow very fast. When you are dealing with non-mutually exclusive classes you should use multitask learning. In this case, the sum of the outputs is no longer 1. In the simplest case, you could think of multitask learning as a shared NN where the last step is made by multiple logistic regression. In more complex cases the last step could be made by a combination of different softmax and logistic regressions.

In conclusion:

• If you need to use non-mutually exclusive classes use multitask learning. Eventually, you will use in multitask learning softmax regression and/or logist regression.
• If you need to use more than two mutually exclusive classes use softmax regression.
• If you need to use only two exclusive classes use logistic regression.