What are possible ways to combat overfitting or improve the test accuracy in my case?

Question

I have asked a question here, and one of the comments suggested that this is a case of severe overfitting. I made a neural network, which uses residual boosting (which is done via a KNN), and I am still just able to get < 50% accuracy on the test set.

What should I do?

I tried everything from reducing the number of epochs to replacing some layers with dropout.

Here is the source code.

Answer 1

There are a few issues you need to address first.

1. Normalise your data. You should try and keep your values for each input in a good range, otherwise you're never going to train anything useful. A simple way of doing this could be to divide each value by the maximum value for that input. This will ensure they are between 0 and 1, or you could divide by the average so you get approximately 1. Whatever the case, you need to do something about decreasing the magnitude of your inputs (also make sure your are consistently scaling your data, especially during validation and training, otherwise your data will likely be misinterpreted by your model)
2. Optional: Augment your data. You have a very small dataset to work with here, if there's any way you can augment this to make it larger, you will likely get much better performance on validation and testing sets. Augmenting is just creating new data that is still representative of the task (as a visual example, you might augment your image dataset by flipping some images and including those in training as well as the unflipped versions)
3. Split your data into Training, Validation and Testing sets. Training is used for what you expect - training. Validation is used to validate your model on unseen data. The reason for the testing set is obviously you are going to try your hardest to tweak and modify the model so it performs best on your validation set, but this incurs a kind of bias as you are only tweaking for this specific set of data, which is not completely representative of the real world. So try your hardest to perform best on the validation set by changing model hyper-parameters, but confirm it's real use on the testing set. As a recommended split for your entire data set, I have had success with 10% testing, 20% validation and 70% training, but you might be better off with a different split.
4. Test different models. Train your network for a small amount of time (say only 10 or so epochs) and see how it performs based on the validation set. Keep in mind because your dataset is particularly small (only ~300 samples in your training set) you don't need to train for many epochs at all to get a good representation of how your model will perform (an epoch being a complete loop over all training data). Make sure when you're doing this you're training in batches.
5. Try different regularization methods. One of the primary ones being good initialization. Make sure you're initializing your parameters with something like Xavier initialization of similar. Other suggestions include: dropout (though you said you already tried this, maybe try different values for dropout), weight decay and batch normalization layers. To be honest though, I think one of the 4 steps above will solve this issue, so you likely won't need to do any of these.

I may have made some points for things you have already done as there was a lot of code for me to look over in your source. But the biggest issue I noticed was that your data wasn't normalised at all and that usually creates many issues, so definitely try that first if you haven't already.