I am building a neural network that takes as input 202 units and outputs a 200 dimension continuous variable. While trying to find the best model, one of the parameters i tune is the the number of hidden units in each layer. I noticed that the training and validation error does not increase despite the increase in number of hidden units and hence weight parameters.
The loss I use is MSE, which outputs the average error incurred for each dimension out of 200 dimensions. Despite increasing the network size from 500x500 to 600x600, the training error hovers around
0.001 ish and the validation error also show similar results.
The 500x500 network however performs slightly better on the test set than the 600x600 model. Since the 600x600 model performed slightly worse on the test set, I would have expected the model to have overfit the training set and achieve a lower training error than the 500x500 model. That is not the case however.
What could be a probable explanation for such a behaviour ? I can't seem to achieve better performance by increasing the number of weights.