2
$\begingroup$

Today I was going through a tutorial of Andrew Ng about Inception network. He said that GoogLeNet's hidden layers are also good in prediction and it had somehow a regularization effect, so it reduces overfitting. I also search on this topic and tried to figure out by reading the GoogLeNet paper. But I am not satisfied.

Can anyone give me any mathematical intuition or reasoning about this in detail?

Improve this question
$\endgroup$
3
  • $\begingroup$ Hi. Please, add a link to the tutorial you were reading or watching. $\endgroup$ – nbro♦ Nov 29 '19 at 21:33
  • $\begingroup$ Course: Convolutional Neural Network, Vendor: deeplearning.ai, Website: coursera.org, Course Timestamp: 2nd week (InceptionNet) $\endgroup$ – Mahir Mahbub Nov 29 '19 at 21:59
  • $\begingroup$ Is this the video? $\endgroup$ – nbro♦ Nov 29 '19 at 22:37
1
$\begingroup$

The main reason of overfitting in any neural network is having too many unrestricted trainable degrees of freedom in the model. Methods similar to dropout reduce the number of neurons at each training run which effectively means having a smaller network. On the other hand in $l_1$ and $l_2$ regularization, a term added to the loss function which put a constraint on the total loss calculated at each run. So what we are trying to minimize with such regularizations is not just $L$, but $L +l_1*f(w)$ (for example).

What I understand from that paper is that the auxiliary outputs do both at the same time: the point is to build smaller version of the same inception network within itself, and use the losses obtained from those as a constraint on the final loss function. The full model described in the paper is essentially 3 separate models: the first one is a network with 3 inception modules, the second one with 6 and the final one has 9. When loss is calculated, results from the auxiliary outputs added to the total loss with a 0.3 weight. Let us write this as follows:

$L = 0.7*L_9+ 0.3*(L_6+L_3). $

Here $L_3$ and $L_6$ represent the losses calculated at first and second outputs, and $L_9$ is the loss calculated at the final output of the network.

This is the function we wish to minimize during training. But when the evaluation is made, the auxiliary outputs are discarded, and just the final softmax layer is used. Not very dissimilar from the idea of using dropout during training but using the full model for predictions.

Improve this answer
$\endgroup$
1
  • $\begingroup$ It might be a good idea to use latex to format your answer. $\endgroup$ – nbro♦ Dec 1 '19 at 2:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.