So I have a deep learning model and three data sets (images). My theory is that one of these data sets should function better when it comes to training a deep learning model (meaning that the model will be able to achieve better performance (higher accuracy) with one of these data sets to serve one classification purpose)
I just want to safe check my approach here. I understand the random nature of training deep learning models and the difficulties associated with such experiment. Though, I want someone who can point out maybe a red flag here.
I am wondering about these things:
do you think using an optimizer with default parameters and repeating the training process, let's say, 30 times for each data set and picking the best performance is a safe approach? I am mainly worried here that modifying the hyperparamters of the optimizer might result in better results for let's say one of the data sets.
what about seeding the weights initialization? do you think that I should seed them and then modify the hyperparameters until I get the best convergence or not seed and still modify the hyper parameters?
I am sorry for the generality of my question. I hope if someone can point me in the right direction.