3
$\begingroup$

Is it possible to exclude specific layers from the optimization?

For example, let's say I have an input layer, 2 hidden layers, and the output layer. I know there is a perfect solution for my problem with this setup and I already know the perfect weights between the first and the second hidden layer.

Can I have the weights between the first and the second hidden layer be fixed during the training phase?

I understand that I could just not update these specific weights after I computed the backpropagation for the entire network. But if I throw away those specific weights, will this affect the optimization of the rest of my weights?

$\endgroup$
4
$\begingroup$

Yes, you can fix (or freeze) some of the weights during the training of a neural network. In fact, this is done in the most common form of transfer learning (which is described here). I don't know exactly how this affects learning in general. In transfer learning, this is definitely beneficial, as we are freezing the weights that are associated with the learned general features of objects, such as corners (where general here is defined intuitively), which can be useful for other tasks, so, by having them frozen, we reuse them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.