I'm building a 5-class classifier with a private dataset. Each data sample has 67 features and there are about 40000 samples. Samples of a particular class were duplicated to overcome class imbalance problems (hence 40000 samples).
With a one-vs-one multi-class SVM, I am getting an accuracy of ~79% on the validation set. The features were standardized to get 79% accuracy. Without standardization, the accuracy I get is ~72%. Similar result when I tried 50-fold cross validation.
Now moving on to MLP results,
Exp 1:
- Network Architecture: [67 40 5]
- Optimizer: Adam
- Learning Rate: exponential decay of base learning rate
- Validation Accuracy: ~45%
- Observation: Both training accuracy and validation accuracy stops improving.
Exp 2: Repeated Exp 1 with batchnorm layer
- Validation Accuracy: ~50%
- Observation: Got 5% increase in accuracy.
Exp 3:
To overfit, increased the depth of MLP. A deeper version of Exp 1 network
- Network Architecture: [67 40 40 40 40 40 40 5]
- Optimizer: Adam
- Learning Rate: exponential decay of base learning rate
- Validation Accuracy: ~55%
Thoughts on what might be happening?