4
$\begingroup$

So I read about softmax from this article. Apparently to me these 2 are almost similar, except that the probability of all classes in softmax add to 1. According to their last paragraph for number of classes = 2, softmax reduces to LR. What I want to know is other than number of classes = 2 what are the essential differences between LR and softmax. Like in terms of:

  • Performance.
  • Computational Requirements.
  • Ease of calculation of derivatives.
  • Ease of visualization.
  • Number of minimas in the convex cost function, etc.

Other differences are also welcome!

EDIT : I am asking for relative comparisons only, so that at the time of implementation I have no difficulty in selecting which method of implementation to use.

$\endgroup$
3
$\begingroup$

As written, SoftMax is a generalization of Logistic Regression.
Hence:

  1. Performance
    If the model has more than 2 classes then you can't compare. Given K = 2 they are the same.
  2. Computation Requirements
    Please explain as the computational requirements require the data, enough memory to hold it and enough time to let run.
  3. Ease of Calculation of Derivatives
    The cost function is summation hence once you do it for one element you do it fol all.
  4. Ease of Visualization
    Well, it is easy to visualize the Confusion Matrix even for K = 10 classes. So no issue here.
  5. Cost Function
    The cost function is convex. Yet not Strictly Convex hence there infinite number of minimas.
$\endgroup$
  • $\begingroup$ Thank You. Where is the Meta forum of AI? I want to suggest to add Deep Learning to the name. I will draw many more users. We must do it before it gets closed. $\endgroup$ – Royi May 11 '18 at 14:10
  • $\begingroup$ ai.meta.stackexchange.com Also feel free to discuss on chat.stackexchange.com/rooms/43371/the-singularity $\endgroup$ – DukeZhou May 11 '18 at 15:21
  • $\begingroup$ All i asked for relative comparison...absolute comparison is not possible $\endgroup$ – DuttaA May 22 '18 at 4:22
1
$\begingroup$

I don't think that it's useful to differentiate logistic regression and softmax based on your terms. This is because you don't choose one or the other based on performance/computational requirements/ease of calculation of derivatives/...

The fact is that you use one or the other based on which is your problem.

If you need to recognize cat pictures vs. non-cat pictures you will use logistic regression (even with a very complex NN the last step will be always a logistic regression). Of course, you could use softmax but the outputs will be redundant, i.e. one output will always be one minus the other.

If you need to recognize cat pictures vs. dog pictures vs. other pictures you will use softmax. Note that, in order to use softmax, you need to have only mutually exclusive classes. Mutually exclusive classes mean that an example cannot belong to multiple classes. What if a picture represents both a dog and a cat? In this case, it should be marked as other picture. If you want to avoid these you could use one more class to denote pictures with both cats and dogs.

However, if you want to recognize cats, dogs, birds, fishes, boats, houses, etc. the number of mixed classes that you need to include will grow very fast. When you are dealing with non-mutually exclusive classes you should use multitask learning. In this case, the sum of the outputs is no longer 1. In the simplest case, you could think of multitask learning as a shared NN where the last step is made by multiple logistic regression. In more complex cases the last step could be made by a combination of different softmax and logistic regressions.

In conclusion:

  • If you need to use non-mutually exclusive classes use multitask learning. Eventually, you will use in multitask learning softmax regression and/or logist regression.
  • If you need to use more than two mutually exclusive classes use softmax regression.
  • If you need to use only two exclusive classes use logistic regression.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.