14

State-of-the-art is a tough bar, because it's not clear how it should be measured. An alternative criteria, which is akin to state-of-the-art, is to ask when you might prefer to try an SVM. SVMs have several advantages: Through the kernel trick, the runtime of an SVM does not increase significantly if you want to learn patterns over many non-linear ...


7

Neural Networks have other short comings as well. It takes much longer and far more resources to train a neural network than something like a random forest. So if you need speed of training or are resource constrained in anyway, you probably should not look at Neural Networks first. Evaluation of a trained deep NN can be much more expensive than competing ...


7

Deep Learning and Neural Networks are getting most of the focus because of recent advances in the field and most experts believe it to be the future of solving machine learning problems. But make no mistake, classical models still produce exceptional results and in certain problems, they can produce better results than deep learning. Linear Regression is ...


6

I find the chapter on machine learning from Russell & Norvig is a pretty good place to start with SVMs. I think this is Chapter 18? One way to understand an SVM is as a kind of neural network, but this is not usually an intuitive approach for a beginner (unless your NN knowledge is already quite good). A better way to understand SVMs is as consisting ...


5

Just to add to what has been said in @MikeWise's brilliant answer, All things equal, deep learning models generally rank supreme when compared to other algorithms as the size of the dataset increases: Like everything, it all boils down to the dataset at hand, neural networks are good on other datasets but at the same time, they will be bad on other ...


5

The most likely explanation is that you're using too many training examples for your SVM implementation. SVMs are based around a kernel function. Most implementations explicitly store this as an NxN matrix of distances between the training points to avoid computing entries over and over again. In your case, with 75% of 700,000 examples, this matrix will ...


3

This is not an answer (I don't have enough reputation to comment). I did something close to this in my master's thesis and think it is close to what you are interested in. In it, I had developed a framework for extracting metadata from web-based educational content. This metadata was used for classifying the the educaitonal content for many different ...


2

It would be hard to tell if you don't provide what kind of data/problem you are working on, but LDA works well when data that are grouped in gaussian blobs surrounding centroids while vanilla SVM works well when the data is almost linearly separable and naive bayes works well when your features are relatively independent of each other.


2

I've summarized the key ideas of SVMs. So this is how $\gamma$ is used with a gaussian Kernel: $$K_{\text{Gauss}}(\mathbf{x}_i, \mathbf{x}_j) = e^{\frac{-\gamma\|\mathbf{x}_i - \mathbf{x}_j\|^2}{2 \sigma^2}}$$ The bigger the $\gamma$, the more "linear" the decision boundary will be. The closer to 0, the more support vectors you have/the more non-linear the ...


2

You can try using a multi-input model. Here is a recent post with a similar discussion, with the required architecture defined in the answer. Instead of combining the separate models, you can create a model which uses image and numerical data side by side. Keras allows you to use different types of data using multi input structure via functional API. And ...


2

First, what makes the neural network different than linear regression is the non-linearity (activation function), not the number of layers. So, a neural network with $n$ layers with no non-linearities is still the same as linear regression. Second, SVM finds the hyperplane of maximum margin. You are not guaranteed to find that hyperplane that has maximum ...


2

The straight dashed-line shows the typical decision line in logistic regression or any linear classifier. The dashed-circle shows the decision line from SVM. Obviously, since the data is not linearly separable in the original 2D feature space, if someone makes a higher dimension space by taking into account non-linear interaction of the original 2 features ...


2

It is important to note that the exact statement is the eqation given below can never be 0 for misclassified points in $ S^+$ $$ E(X) = (y - \text{sign}\{\overline{W} \cdot \overline{X}\}) $$ And $S+$ is defined as the set of all misclassified training points $X \in S$ that satisfy the condition $y(\overline{W} \cdot \overline{X})<0 $ which means that $y$ ...


2

That is the hinge loss, a type of loss most notably used for SVM classification. The hinge loss is typically defined as: $$ \ell(y)=\max (0,1-t \cdot y), $$ which, in your use case, is something like this: $$ \operatorname{cost}\left(h_{\theta}(x), y\right)=\left\{\begin{array}{ll} \max \left(0,1-\theta^{T} x\right) & \text { if } y=1 \\ \max \left(0,1+\...


2

Smoothness here is the mathematical definition, so as you implied smoothness is ruled out by output data with sharp spikes or discontinuous jumps (and possibly the data of the gradient, the gradient's gradient, ad infinitum, depending on who defines smoothness). By any definition a lot of activation functions are not smooth, for example RELU. This means ...


2

The hinge loss/error function is the typical loss function used for binary classification (but it can also be extended to multi-class classification) in the context of support vector machines, although it can also be used in the context of neural networks, as described here. The hinge loss function is defined as follows $$ \ell(y) = \max(0, 1-t \cdot y) \tag{...


1

In the least-squares SVM (LS-SVM) the non-zero Lagrange multipliers ($\alpha$) are the support values. The corresponding data points are the support vectors. Johan Suykens explains this in Least Squares Support Vector Machines.


1

My question is, can I rely on my Accuracy (mean & standard deviation) for future games even though my Testing Accuracy is lower than 52.5%? If by Accuracy you mean training accuracy, then absolutely you should not trust those values. For almost all machine learning algorithms there is a problem with overfitting to training data, which will result in ...


1

A kernel function $f : \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ is a valid support vector kernel if it is a Mercer kernel. Mercer's condition essentially ensures that the Gram matrix of the kernel is positive semi-definite. Interestingly, this ensures that the SVM objective is convex. The Euclidean distance function does not satisfy Mercer's ...


1

So actually I managed to get hold of my lecturer to explain the argmax to argmin conversion. Generally speaking maximising $\frac{1}{||w||}$ is identical to minimising $||w||$. As $||w||$ in $\frac{1}{||w||}$ decreases, the overall value increases, i.e. we maximise it. The reason for choosing $\frac{1}{2}||w||^2$ turns out to be a less mathematic and more ...


1

SVM complexity is $O(\max(n,d)\min(n,d)^2)$ according to Chapelle, Olivier. "Training a support vector machine in the primal." Neural Computation 19.5 (2007): 1155-1178. $n$ is the number of instances and $d$ is the number of dimensions. I'm assuming that you have more instances than dimensions giving a complexity of $O(nd^2)$. Hopefully this ...


1

$\mathbf{x} \in \mathbb{R}^p$ and $\mathbf{x}' \in \mathbb{R}^p$ are two inputs (or feature vectors). In the context of classification with an SVM, you are given a dataset $D = \{(\mathbf{x}_i, y_i) \}_{i=1}^N$, where $\mathbf{x}_i \in \mathbb{R}^p$ is an input (or point) and $y_i$ the corresponding label. The goal is to find a hyperplane that classifies ...


1

Bag-of-visual words (BOVW) was classicly used in computer vision before the introduction of neural networks or some more advanced classical techniques, such us VLAD or Fisher Vectors. In any case, it is a good technique to use, but it is not the state-of-the-art today, and I won't recommend you use it for a real-life project.


1

Based on this repository: https://github.com/arkm97/svm-from-scratch/blob/master/SVM_from_scratch.ipynb I will try to reverse engineering that concept: So firstly there is issue for DataCleaning(removing 0,values, serialize, normalize) normalizing dataset (replaced by its difference from the mean) credit_df_norm = (credit_df - credit_df.mean())/(credit_df....


1

The usage of the word "kernel" in the context of support vector machines probably comes from its usage in the context of integral transforms. See the article Kernel of an integral operator, and the questions What is the difference between a kernel and a function? and Why is the kernel of an integral transform called kernel?. The word "kernel" has been ...


1

I will try to give you a simplified explanation of how SVMs work. The data one works with can be of two types. Either it is very easily separable and there is a clear straight line boundary between data points of different classes. We call such data linearly separable(image) or the data points of the classes are mixed in a way that there is no clear ...


1

To quickly train the SVM , you can try to Use Linear SVM or Use scaled data. sources: https://www.researchgate.net/publication/2926909_A_Practical_Guide_to_Support_Vector_Classification_Chih-Wei_Hsu_Chih-Chung_Chang_and_Chih-Jen_Lin


1

I think you should use a linear kernel, 'cause training SVM with a linear kernel is faster than with another kernel, especially for text classification. Good luck https://www.svm-tutorial.com/2014/10/svm-linear-kernel-good-text-classification/


1

Rougly speaking, the higher the gamma, the more complex the model, the higher the risk of overfitting. In fact, as you can read on the page you linked: If gamma is too large, the radius of the area of influence of the support vectors only includes the support vector itself[...] When gamma is very small, the model is too constrained and cannot ...


Only top voted, non community-wiki answers of a minimum length are eligible