0
$\begingroup$

Now, the main idea behind ResNet is to enable the network to learn the residual value needed to modify the input for achieving the best result. In other words, given an input x, the output y is computed as y = F(x) + x, and the goal is to learn the function F (the filter). If the ideal output is the identity function, it implies that the model has discovered the best filter to be all zeros. This situation could arise due to bad initialization, resulting in the best filter being all zeros. Alternatively, it could be the optimal solution if there are no more residual blocks after it. If this is not the optimal solution, we can reinitialize the filters and retrain them.

I would greatly appreciate any guidance or corrections.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ I don't understand how could you tell if this is the optimal solution and leave it as it is; or if it is not the optimal solution and then re-init the weights ? It may be that one of the first res blocks learns to be an identity function instead of the last block. I mean, it does not have to be the last block. $\endgroup$
    – pi-tau
    Aug 24 at 19:36

1 Answer 1

Reset to default
2
$\begingroup$

I think what is done in practice is that you just train multiple networks with different random seeds. This greatly reduces the possibility for getting bad models due to bad initialization.

At the end you can just work with the model that shows the best performance on the validation set, or instead, use all the models as an ensemble.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .