How to compare the training performance of a model on different data input?

Question

So I have a deep learning model and three data sets (images). My theory is that one of these data sets should function better when it comes to training a deep learning model (meaning that the model will be able to achieve better performance (higher accuracy) with one of these data sets to serve one classification purpose)

I just want to safe check my approach here. I understand the random nature of training deep learning models and the difficulties associated with such experiment. Though, I want someone who can point out maybe a red flag here.

I am wondering about these things:

1. do you think using an optimizer with default parameters and repeating the training process, let's say, 30 times for each data set and picking the best performance is a safe approach? I am mainly worried here that modifying the hyperparamters of the optimizer might result in better results for let's say one of the data sets.

2. what about seeding the weights initialization? do you think that I should seed them and then modify the hyperparameters until I get the best convergence or not seed and still modify the hyper parameters?

I am sorry for the generality of my question. I hope if someone can point me in the right direction.

Answer 1

Well, this is more of a subjective question but I will give my best shot.

Regarding your 1st question, the nature of methods in deep learning states that you should experiment otherwise you will only have weak intuitions. So which dataset should you choose? I would say, select your primary concerns since you cannot try everything. If it's time, make a little random or grid search over hyperparameters and observe speed of convergence in the datasets then choose the best one. If it's accuracy, ideally you should make an input analysis on distribution of data, etc. If you can analyze it well, you are expected to obtain best result from the chosen dataset. So training 30 epochs each dataset and choosing the one with lowest loss is not a safe approach. Maybe your model will converge at 40th epoch for the worst dataset in the 30th epoch but it will be much more robust. My final advice is to set a threshold for your evaluation metrics, once you reach them in any of your dataset choose that one -assuming datasets are more or less equal in number of instances. In this way you are at least know that the dataset you choose satisfies your expectations.

For your 2nd question, seeding during comparisons is a good approach though in the long run it won't matter much. Hence there is no harm in seeding unless you are very very very unlucky.