2
$\begingroup$

In general, how does one balance the two opposing forces of allowing a layer of a neural network to adapt/learn to its training data, while also forcing the neural network to represent some known structure (e.g., representing a family of functions)? The neural network might find the optimal weights, but those weights might no longer make the layer represent the function I originally intended.

For example, suppose I want to create a convolutional layer in the middle of my neural network that is a low-pass filter. In the context of the entire network, however, the layer might cease to be a low-pass filter at the end of training because the backpropagation algorithm found a better optimum. How do I allow the weights to be as optimal as possible, while still maintaining the low-pass characteristics I originally wanted?

Thanks in advance for any help and suggestions. General tips or pointing to specific literature would be much appreciated.

$\endgroup$
  • 2
    $\begingroup$ Two ways to do it - freeze above mentioned layer, don't update it. Another - add regularizer to loss wich force that layer to retain it's properties. Latter could be very non-trivial $\endgroup$ – mirror2image Jun 11 at 5:34
1
$\begingroup$

Extending @mirror2image's comment, if you have a certain metric that allows you to measure how close the intended layer is to a low pass filter (something that compares its output with what a low pass filter would have produced, for example), the simplest way to achieve what you want would be to add a term in your loss function that calculates the value of this metric. This way, each time you do a training step, the network now is not only made to output the correct predictions but is also forced to do so while also keeping that specific layer's behavior as close to a low-pass filter as possible. This is the most common way of tweaking the behavior of neural networks and is often encountered in many research papers.

$\endgroup$
1
$\begingroup$

the idea about training that it is allowed for weights (not layer as you wrote) "learn" values what they "want" from general network setting. But actualy i v thinked about that too, and to have weights represent as you want, you can first train a shorter network, while having those weights (1) as very last, so you have maximum control over them, after trained a lot append next layers to those and learning rate for weights (1) make much smaller as for new weights

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.