Why are traditional ML models still used over deep neural networks?

Question

I'm still on my first steps in the Data Science field. I played with some DL frameworks, like TensorFlow (pure) and Keras (on top) before, and know a little bit of some "classic machine learning" algorithms like decision trees, k-nearest neighbors, etc.

For example, image classification problems can be solved with deep learning, but some people also use the SVM.

Why are traditional ML models still used over neural networks, if neural networks seem to be superior to traditional ML models? Keras is rather simple to use, so why don't people just use deep neural networks with Keras? What are the pros and cons of each approach (considering the same problem)?

Answer 1

Why are still traditional machine learning (ML) models used over neural networks if neural networks seem to be superior to traditional ML models?

Of course, the model that achieves state-of-the-art performance depends on the problem, available datasets, etc., so a comprehensive comparison between traditional ML models and deep neural networks is not appropriate for this website, because it requires a lot of time and space. However, there are certain disadvantages of deep neural networks compared to traditional machine learning models, such as k-nearest neighbors, linear regression, logistic regression, naive Bayes, Gaussian processes, support vector machines, hidden Markov models and decision trees.

• Often, traditional ML models are conceptually simpler (for example, k-NN or linear regression are much simpler than deep neural networks, such as LSTMs).

• Personally, I've noticed that traditional ML models can be used more easily compared to deep neural networks, given the existence of libraries, like scikit-learn, which really have a simple and intuitive API (even though you apparently do not agree with this).

• Deep neural networks usually require more data than traditional ML models in order not to overfit. Empirically, I've once observed that certain traditional ML models can achieve comparable performance to deep neural networks in the case of small training datasets.

• Even though there's already a new and promising area of study called Bayesian deep learning, most deep neural networks do not really provide any uncertainty guarantees, they only provide you a point estimate. This is a big limitation, because, in areas like healthcare, uncertainty measures are required. In those cases, Gaussian processes may be more appropriate.

Answer 2

This question is very broad, so let me attempt to answer it using my own background in time series analysis.

As an example, why would I continue using ARIMA to forecast a time series? Why not simply use an LSTM model by default, since this is a type of recurrent neural network that takes time-related dependencies into account?

Well, an LSTM model is not good at modelling all time series. It is effective when it comes to modelling volatile data, but ARIMA still outperforms when it comes to forecasting trend data - LSTM tends to overemphasise volatile patterns in future predictions.

Let's take an example of forecasting weekly hotel cancellations by potential customers. The second time series shows much more variability in the number of weekly hotel cancellations than the first:

H1 Time Series

H2 Time Series

Based on MDA (mean directional accuracy), RMSE (root mean squared error), and MFE (mean forecast error) - ARIMA demonstrates superior performance overall for the first time series, while LSTM shows better performance for the second:

On the basis of this example - which is quite specific given the broadness of your question - deep learning techniques are not always used because simpler models can perform better under certain circumstances. It is all about understanding the data you are working with and then choosing the model - not the other way around.

Answer 3

SVM is generally considered objectively better than deep learning for standard machine learning tasks.

SVM or decision trees.

Deep Learning is beneficial when there is structure in the data that can't be easily represented by some type of kernel.

I'm actually really interested in why decision trees haven't been used for computer vision in conjunction with deep learning feature extraction.