1
$\begingroup$

After some time starting the deep learning project, training output files (model weights,training configuration files) will be piled up. Naming all outputs and training files can become complicated if the clean naming convention is not used. There are some example naming styles below. I wonder that how do you manage your training outputs and training files?

'Outputs/run1/'
'Outputs/run2/'
'Outputs/run3'
    .
    .
    .

'Outputs/20230222/'
'Outputs/20230223/'
'Outputs/20230224/'
    .
    .
    .

'Outputs/WithDropout/'
'Outputs/WithDropout_RotationTransform/'
'Outputs/WithDropout_RotationTransform_AdamOptimizer/'
    .
    .
    .
Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

Depending on how many instances of models you train you can do one of the following:

  1. For when the amount of models is still somewhat manageable: Generate a settings file together with the model file in which you store all the hyperparameters of the model.
  2. For when it really gets out of hand with the amount of models. Generate a random unique number which you can use to name the model, and store the settings and the file name of all models in a csv document. You can then use the csv document to retrieve the correct model name corresponding to a set of settings and a result.

It's a bit of a hassle to implement but its worth it in the end ;) You can of course also mix and match the options if that suits your needs better. Unfortunately, I do not know a simple 'hack' which allows you to do this very easily.

You can also try to add the model parameters in the name of the model itself, but in my experience this usually gets messy real fast once you realise 'oh i have to add this parameter as well', and 'oh this model does not have this parameter, but the other one does'.

If you do something like bayesian optimization, a service such as WeightsAndBiases can keep track of all this stuff for you. The applicability of such a method is of course heavily depend

Improve this answer
$\endgroup$
1
  • $\begingroup$ I create folders for each training and store training parameters in yaml files in those folders. I add Tensorboard event files in those folders too. When I visualize the event files in the Tensorboard it becomes difficult to interpret graphs as there are many training graphs. So I wondered how people manage and interpret many training graphs easily. It could be easy if I could associate the training names with the graphs clearly. Thanks for your answer :) $\endgroup$
    – Ugurcan
    Feb 23 at 6:55
0
$\begingroup$

You could use the most important distinctions to build a folder structure. For example I experiment with multiple model architectures (resnet, mobilenet, ...) and different types of classification (binary, multiclass and multilabel). Later in the development process it is possible new classes are added. So these parameters I include into the folder structure as well. Eventually I use the date like you showed in your example.

For example: models/resnet/multiclass_4/20230302/...

This would be a resnet multiclass model with 4 classes saved 2 march 2023.

After I save the model I gather all the relevant training outputs to save them in a separate json file in the model directory. Finally I add a custom field called "description" where I give a short explanation what I tried compared to the previous training cycle.

Disadvantages:

  • The description field will not help if you compare models which were not trained in succession.
  • You can't get a lot of information based on the folder structure.

Advantage:

  • No endless series of subdirectories or chaotic directory names.
  • Minimal information in description can help you remember which model it was.
Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .