I've trained my artificial neural network, and, as per standard practice, I've picked out the one neural network throughout training that did the best on my validation dataset. That is, the neural network learned from the training data, and generalized to the validation data.
However, when I run the neural network on the test data, it performs poorly. What should I do next?
From my understanding of the theoretical framework, the goal of validation is to ensure that that the network's parameters don't overfit to the training set. (If they do, we'll detect it because the validation score will be bad.) However, the goal of an additional test dataset set beyond the validation dataset is to ensure that our hyperparameters don't overfit. In most scenarios, we train multiple models with different learning rates, etc., and pick the one that does the best on the validation dataset. However, we might just be cherry picking the one that does the best for that validation dataset and doesn't generalize to a test set. So, we add an extra test set to detect if that happens.
My question is about an analogous case, except for that my hyperparameter is just the training step of the model. I picked out the model that's checkpointed as having the highest validation score. But, when I run it on the test set, it does poorly, showing that I've cherry picked the model with the highest validation score but it still doesn't generalize.
What do I do next in this scenario? Do I just follow the same advice from these questions about overfitting, or is this a special case because the model does seem to generalize to the validation dataset?
(Note: this question is different than regular overfitting because it's about hyperparameters overfitting the validation set, not regular parameters overfitting the train set. I've also looked at these guidelines but they don't seem to apply to this more general theoretical question.)