Does it make sense for a logistic regression model to perform better than a neural network on the Iris data set?

Question

Per a review post, a simple Logistic Regression model on the Iris data set gets about 97% test accuracy on iris dataset whereas a neural network gets just 94%. The neural network model used in Keras is

model = tf.keras.Sequential([
    tf.keras.layers.Dense(500, input_dim=4, activation='relu'),
    tf.keras.layers.Dense(256, activation='relu'),
    tf.keras.layers.Dense(256, activation='relu'),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(3, activation='softmax')
])
model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

The model is fit for 30 epochs using a batch size of 20.

Note that I did try fewer neurons and layers but none of them got better performance.

Does this make sense? Can any other neural network get a higher test accuracy than a logistic regression model?

Answer 1

The most likely explanation is that the neural network model proposed has a much higher model capacity than the logistic regression model.

In fact, the neural network used in the code has 246,343 trainable parameters; the full Iris data set has only 150 samples and only four features – so the model is much more complex than the training data. A neural network model with fewer neurons or layers will likely generalise much better. A logistic regression model is much simpler and has far fewer parameters, so it is in some sense forced to "learn" better.

The neural network may simply overfit to the training data as the capacity of the model is large enough; an analogous idea is trying to fit a line to data, then trying to fit a degree 100 polynomial. While the polynomial may fit the training data better, it is less likely to generalise well.

The hyperparameters used in the training of the network might also not be optimal; I haven't experimented, but neural network training can be a little sensitive to the choices made during training.

Answer 2

Yes, it absolutely makes sense as the Iris dataset is linearly separable (in the sense that linear decision boundaries are near optimal). This can just about be seen in the scatter plots:

Particularly look at petal width versus petal length, two linear decision boundaries for that pair of variables gives a very low error rate already. This shouldn't be surprising as the dataset was introduced in a paper by Fisher describing his linear discriminant method (about 1936?).

As @htl (+1) rightly points out, a large neural network has way too much capacity for this problem and will likely overfit. However you will probably find that a small neural network fares no better than a linear classifier for this problem.

There is a lot of excitement about deep neural networks over the last few years, but they are no better than most existing classifiers for these sorts of benchmark dataset. They are best (IMHO) for problems with very large datasets, or where convolutional layers are useful.

The comp.ai.neural-nets (an old Usenet news group that we used before the WWW - I trained my first neural net back in 1990) FAQ is well worth a read for anyone wanting to experiment with neural networks. It contains a lot of folk wisdom that is good to know. From part 3:

Subject: How many hidden layers should I use?

You may not need any hidden layers at all. Linear and generalized linear models are useful in a wide variety of applications (McCullagh and Nelder 1989). And even if the function you want to learn is mildly nonlinear, you may get better generalization with a simple linear model than with a complicated nonlinear model if there is too little data or too much noise to estimate the nonlinearities accurately.

This is excellent advice.

BTW I this case, you might want to try and diagnose the problem by training your neural network on a two-dimensional version of the problem, so that you can plot the decision boundary of the model and see what it is doing. Sepal Width -v- Sepal Length might be a good choice. Experimenting with low dimensional problems that can be directly visualised is often very informative. I Often use Brian Ripley's synthetic dataset in my work because you can easily generate as much data as you could want.