2
$\begingroup$

I'm building a 5-class classifier with a private dataset. Each data sample has 67 features and there are about 40000 samples. Samples of a particular class were duplicated to overcome class imbalance problems (hence 40000 samples).

With a one-vs-one multi-class SVM, I am getting an accuracy of ~79% on the validation set. The features were standardized to get 79% accuracy. Without standardization, the accuracy I get is ~72%. Similar result when I tried 50-fold cross validation.

Now moving on to MLP results,

Exp 1:

  • Network Architecture: [67 40 5]
  • Optimizer: Adam
  • Learning Rate: exponential decay of base learning rate
  • Validation Accuracy: ~45%
  • Observation: Both training accuracy and validation accuracy stops improving.

Exp 2: Repeated Exp 1 with batchnorm layer

  • Validation Accuracy: ~50%
  • Observation: Got 5% increase in accuracy.

Exp 3:

To overfit, increased the depth of MLP. A deeper version of Exp 1 network

  • Network Architecture: [67 40 40 40 40 40 40 5]
  • Optimizer: Adam
  • Learning Rate: exponential decay of base learning rate
  • Validation Accuracy: ~55%

Thoughts on what might be happening?

Improve this question
$\endgroup$
4
  • $\begingroup$ Such results are not unexpected. 67 dim vector is small, multiclass SVM would likely produce better results then NN. NN start to shine with dimensionality >=1000. If you actually want overfit, just for fun, you probably need bigger network with 128 to 512 layer size, at least 3 layers (not counting softmax), one 1024 layer may work too. NN should have wide hidden layers, at least in the bottom. $\endgroup$
    – mirror2image
    Dec 28, 2019 at 8:13
  • $\begingroup$ Does your deep layer (exp 3) have skip connections / residual connections? If not, then the gradient is likely vanishing, and early layers aren't learning anything much useful $\endgroup$
    – user3667125
    Dec 22, 2020 at 1:11
  • $\begingroup$ I would use SGD over Adam, as my experience with Adam on small network and small dataset is that it finds a worse optimum than SGD. Additionally, what batch size are you using for neural network? It affects the optimum as well. Smaller gives better generalization, so try something small like 8. $\endgroup$
    – user3667125
    Dec 22, 2020 at 1:13
  • $\begingroup$ Would be helpful to know if exp1/exp2 is overfitting the data more so than exp3. What is both the training accuracy and test accuracy for exp1, exp2, and exp3? $\endgroup$
    – user3667125
    Dec 22, 2020 at 1:16

2 Answers 2

Reset to default
0
$\begingroup$

I guess you are using linear activation functions, maybe you are not initializing your weights, or you are regularizing your model enough.

Initialize weights with glorot, insert dropout layers in between, use Relu as your activation function, stop the training process based on Early Stopping. and just experiment with one hidden layer.

side note: if you use Adam, don't mess with the learning rate.
in SGD optimizer you could use decay because there is a single learning rate for all weight updates and the learning rate does not change during training.
In Adam, the learning rate is maintained for each network weight (parameter) and separately adopted as learning unfolds.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I'm using pytorch to model MLP. I'm assuming weight initialization is handled by pytorch. Also, I'm using ReLU and regularization is 0 by default in pytorch. Also, what sense does it make to use dropout layer when I'm not able to overfit on train data? I'll look into the learning rate aspect you mentioned. $\endgroup$
    – Ashwin Kannan
    May 6, 2018 at 16:34
0
$\begingroup$

For the multiclass SVM, there will be an ensembling effect since you are learning 5*4=20 1vs1 classifiers. It could be an interesting experiment to try the same thing with simple neural networks. Also, since you are standardizing the inputs you could try tanh activations on the first layer after the input. I presume you are using softmax on the output layer.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .