I'm building a 5-class classifier with a private dataset. Each data sample has 67 features and there are about 40000 samples. Samples of a particular class were duplicated to overcome class imbalance problems (hence 40000 samples).

With a one-vs-one multi-class SVM, I am getting an accuracy of ~79% on the validation set. The features were standardized to get 79% accuracy. Without standardization, the accuracy I get is ~72%. Similar result when I tried 50-fold cross validation.

Now moving on to MLP results,

Exp 1:

  • Network Architecture: [67 40 5]
  • Optimizer: Adam
  • Learning Rate: exponential decay of base learning rate
  • Validation Accuracy: ~45%
  • Observation: Both training accuracy and validation accuracy stops improving.

Exp 2: Repeated Exp 1 with batchnorm layer

  • Validation Accuracy: ~50%
  • Observation: Got 5% increase in accuracy.

Exp 3:

To overfit, increased the depth of MLP. A deeper version of Exp 1 network

  • Network Architecture: [67 40 40 40 40 40 40 5]
  • Optimizer: Adam
  • Learning Rate: exponential decay of base learning rate
  • Validation Accuracy: ~55%

Thoughts on what might be happening?

  • $\begingroup$ Welcome to AI! (If you don't get a response over here, you might check the Cross Validated or Data Science forums.) $\endgroup$ – DukeZhou May 7 '18 at 17:43
  • $\begingroup$ Such results are not unexpected. 67 dim vector is small, multiclass SVM would likely produce better results then NN. NN start to shine with dimensionality >=1000. If you actually want overfit, just for fun, you probably need bigger network with 128 to 512 layer size, at least 3 layers (not counting softmax), one 1024 layer may work too. NN should have wide hidden layers, at least in the bottom. $\endgroup$ – mirror2image Dec 28 '19 at 8:13
  • $\begingroup$ Does your deep layer (exp 3) have skip connections / residual connections? If not, then the gradient is likely vanishing, and early layers aren't learning anything much useful $\endgroup$ – user3667125 Dec 22 '20 at 1:11
  • $\begingroup$ I would use SGD over Adam, as my experience with Adam on small network and small dataset is that it finds a worse optimum than SGD. Additionally, what batch size are you using for neural network? It affects the optimum as well. Smaller gives better generalization, so try something small like 8. $\endgroup$ – user3667125 Dec 22 '20 at 1:13
  • $\begingroup$ Would be helpful to know if exp1/exp2 is overfitting the data more so than exp3. What is both the training accuracy and test accuracy for exp1, exp2, and exp3? $\endgroup$ – user3667125 Dec 22 '20 at 1:16

I guess you are using linear activation functions, maybe you are not initializing your weights, or you are regularizing your model enough.

Initialize weights with glorot, insert dropout layers in between, use Relu as your activation function, stop the training process based on Early Stopping. and just experiment with one hidden layer.

side note: if you use Adam, don't mess with the learning rate.
in SGD optimizer you could use decay because there is a single learning rate for all weight updates and the learning rate does not change during training.
In Adam, the learning rate is maintained for each network weight (parameter) and separately adopted as learning unfolds.

  • $\begingroup$ I'm using pytorch to model MLP. I'm assuming weight initialization is handled by pytorch. Also, I'm using ReLU and regularization is 0 by default in pytorch. Also, what sense does it make to use dropout layer when I'm not able to overfit on train data? I'll look into the learning rate aspect you mentioned. $\endgroup$ – Ashwin Kannan May 6 '18 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.