0
$\begingroup$

I'm newbie in Convolutional Neural Networks and I have discovered (and I hope I'm right) that kernels in convolutional layers are learned while training.

If I have a kernel that it is very good to extract the features that I want to extract.

Is there any way to set it to the convolutional layers and don't let the network to modify it?

Maybe, in that case, I have to use something different from a CNN.

$\endgroup$
  • $\begingroup$ Very likely yes, but it depends on the framework you are using whether possible and how to do it. As that is a more practical question involving code examples - you should explain what framework you are using plus maybe show your CNN build code so that answer can show you using your own variables & style - I suggest asking that in Data Science SE, which has more focus on practical how to questions on specific frameworks $\endgroup$ – Neil Slater Jan 14 at 15:38
  • $\begingroup$ On this stack, we could answer "yes" and explain how to do this theoretically - i.e. in short, just don't update weight parameters that you don't want to change - but that might not be enough for you? $\endgroup$ – Neil Slater Jan 14 at 15:39
  • $\begingroup$ @NeilSlater Ok. Thanks a lot for your comment and suggestions. $\endgroup$ – VansFannel Jan 14 at 15:40
  • $\begingroup$ @NeilSlater Could you please how you will do it theoretically? Now, I'm using Python, Tensorflow and Keras. But I don't rule out developing my own implementation of a convolutional network (or maybe not). $\endgroup$ – VansFannel Jan 14 at 15:42
2
$\begingroup$

In most modern neural network frameworks, the update rules for training can be selectively applied to some parameters and not others.

How to do that is dependent on the framework. Some will have the concept of "freezing" a layer, preventing parameters in it being updated. Keras does this for example. Others will do the opposite and expect you to provide a list of trainable parameters - these typically come with helpers that will list all parameters in a neural network, so you would need to add some kind of filter after collecting that data to exclude your pre-trained layer. PyTorch does this (although the linked example is slightly more complex in that it stops calculating gradients too).

If your framework of choice does not allow you to select and isolate layers in the training process, then you still have a couple of options:

  • You could store a copy of layer parameters that you want to keep and force your learning network to re-load these parameters after each mini-batch. This does depend on you having a method that can selectively set parameters.

  • If your pre-trained layers are the first ones, immediately next to the input, then instead of including them in your learning network model, you can pre-process all your training data with just the fixed layers (build a model using only those layers), save the output and use that as an alternative input for the learning layers (build a second model with only the learning layers). Later, once training is complete, you can build a combined neural network out of the fixed layers and the learning layers.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.