Why does training an SVM take so long? How can I speed it up?

Question

I'm trying to create and test non-linear SVMs with various kernels (RBF, Sigmoid, Polynomial) in scikit-learn, to create a model which can classify anomalies and benign behaviors.

My dataset includes 692703 records and I use a 75/25% training/testing split. Also, I use various combinations of features whose dimensionality is between 1 and 14 features. However, the training processes of the various SVMs take much too long. Is this reasonable?

I have also examined the ensemble BaggingClassifier in combination with non-linear SVMs, by configuring the n_jobs parameter to -1; nevertheless, the training process proceeds again too slowly.

How can I speed up the training processes?

Answer 1

The most likely explanation is that you're using too many training examples for your SVM implementation.

SVMs are based around a kernel function. Most implementations explicitly store this as an NxN matrix of distances between the training points to avoid computing entries over and over again.

In your case, with 75% of 700,000 examples, this matrix will require approximately 250GB of RAM to store, which is more than you're likely to have in consumer hardware.

If your SVM implementation can avoid caching the values, you might get a speedup that way, or you might not (you'll waste a lot of time recomputing them).

A much better way to deal with this is to just not use all of the data, since most of it will be redundant from the SVM's perspective (it only benefits from having more data near the decision boundaries). A good starting place would be to randomly discard 90% of the training data, and see what performance looks like.

Answer 2

I think you should use a linear kernel, 'cause training SVM with a linear kernel is faster than with another kernel, especially for text classification. Good luck

https://www.svm-tutorial.com/2014/10/svm-linear-kernel-good-text-classification/

Answer 3

To quickly train the SVM , you can try to Use Linear SVM or Use scaled data.

sources: https://www.researchgate.net/publication/2926909_A_Practical_Guide_to_Support_Vector_Classification_Chih-Wei_Hsu_Chih-Chung_Chang_and_Chih-Jen_Lin

Answer 4

Because you use various combinations of features whose dimensionality is between 1 and 14 features, You might try to use Linear SVM (linear Kernels) would be good for your problem. You could try LIBLINEAR library but the Data should be linearly separable, otherwise test accuracy would be very low.

Answer 5

You can speed up the training time by doing several steps:

1. scale the values of your features
2. use only a limited number of features because this will affect the training time; i.e. when you use 14 features, it means your model has 14 dimensions and it makes computation more complex and take much time.
3. choose a proper kernel, linear SVM kernel usually give the fastest result

Answer 6

The non-linear kernel SVMs can be slow if you have too many training samples. This is due to the fact that the algorithm creates an NxN matrix as @John Doucette answered.

Now there are a few ways to speed up the non-linear kernel SVMs:

• Use the SGDClassifier instead and provide proper parameters for loss, penalty etc. to make it behave like an SVM. The optimisation process is different than libsvm though.
• Use a kernel approximator like Nystroem
• Since you are yourself trying out feature combinations, a Linear SVM can also be good and fast :)

Answer 7

SVM scales rather badly with the number of training samples - from $O(n^2)$ to $O(n^3)$ as told in this answer https://stackoverflow.com/questions/16585465/training-complexity-of-linear-svm.

The vanilla approach requires inversion of $n \times n$ matrix, which is $O(n^3)$ operations in general.

As suggested in the other answers, the most apparent way to reduce the computational and storage complexity is the reduction of number of training samples.

I am even surprised that all this data fits into the memory.