1
$\begingroup$

Suppose that we have different animals that we have 4 types of dogs that we want to detect (Golden Retriever, Black Labrador, Cocker Spaniel, and Pit Bull). The training data consists of png images of a data set of dogs along with their annotations. We want to train a model using YOLOv3. Does the choice of optimizer really matter in terms of training the model? Would the Adam optimizer be better than the Adadelta optimizer? Or would they all basically be the same?

Would some optimizers be better because they allow most of the weights to achieve their "global" minima?

$\endgroup$
1
$\begingroup$

First you should split your data into train and test sets. the test set is the one in which the model will not see it in the training phase. Then train your model with validation and train sets, and evaluate it with test sets. if validation accuracy and test accuracy was close together, your model is working on unseen data. Otherwise, your model is underfitted or overfitted, depends on loss values you can understand and solve it.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Does the choice of optimizer really matter in terms of training the model?

Yes.

Would the Adam optimizer be better than the Adadelta optimizer?

Yes. (But sometimes, adadelta gives better result. Depends upon the dataset and fine-tune mechanism)

Would they all basically be the same?

No

Explanation

Would some optimizers be better because they allow most of the weights to achieve their "global" minima?

It's not possible to check whether the model achieved global minima or not practically.

We can evaluate the model by either over-fitting or under-fitting using train and val set and generalisation of the model with test set. (@sahatib's comment)

| improve this answer | |
$\endgroup$
0
$\begingroup$

I have experimented with this to a small degree and have not noticed that much of an impact. To date Adam appears to give the best results on a variety of image data sets. I have found that "adjusting" the learning rate during training is an effective means of improving model performance and has more impact than selection of the optimizer. Keras has two callbacks that are useful for this purpose. Documentation is at https://keras.io/callbacks/. The ModelCheckpoint callback enables you to save the full model or just the model weights based on monitoring a metric. Typically you monitor validation loss and set the parameter save_best_only=True to save the results for the lowest validation loss. The other useful callback is ReduceLROnPlateau which allows you to adjust the learning rate based on monitoring a metric. Again the metric usually monitored is the validation loss. If the loss fails to reduce after a user set number of epochs (parameter patience) the learning rate will be adjusted by a user set factor(parameter factor). You can think of the training process as travelling down a valley. As you near the bottom the valley it becomes more and more narrow. If your learning rate does not adjust to the "narrowness" there is no way you will get to the bottom of the valley. You can also write a custom callback to adjust the learning rate.I have done this and created one which first adjust the learning rate based on monitoring the training loss until the training accuracy reaches 95%. Then it switches to adjust the learning rate based on monitoring the validation loss. It saves the model weights for the lowest validation loss and loads the model with these weight to make predictions. I have found this approach leads to faster training and higher accuracy. Fact is you can't tell if your model has converged on a global minimum or a local minimum. This is evidenced by the fact that unless you take special efforts to inhibit randomization you can get different results each time you run your model. The loss can be envisioned as a surface in N space where N is the number of trainable parameters. Lord knows what that surface is like and where your initial parameter weights put you on that surface plus how other random processes cause you to traverse that surface. As an example I ran a model at least 20 times and got resultant losses that were very close to each there. Than I ran it again and got far better results for exactly the same data.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.