Recently, I extend a master's thesis. I am now in a training phase for the model associated with it. I have access to many node GPUs. I would like to train this model on different scenarios, e.g. learning rate between 0.1 and 0.001, with batch size 16, 32, 64 or 128, and so on. At the beginning, I have thought using docker, creating a training container for each possible scenario and using airflow to centralize the train results, but I think this is over-killed. Another possibility would have been to use MLFlow with Hydra, but still not sure about it.
What is the best solution to train the model on different scenarios and gather the training performance results in one single place?