2
$\begingroup$

In general, how does one make a neural network learn the training data while also forcing it to represent some known structure (e.g., representing a family of functions)?

The neural network might find the optimal weights, but those weights might no longer make the layer represent the function I originally intended.

For example, suppose I want to create a convolutional layer in the middle of my neural network that is a low-pass filter. In the context of the entire network, however, the layer might cease to be a low-pass filter at the end of training because the backpropagation algorithm found a better optimum.

How do I allow the weights to be as optimal as possible, while still maintaining the low-pass characteristics I originally wanted?

General tips or pointing to specific literature would be much appreciated.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Two ways to do it - freeze above mentioned layer, don't update it. Another - add regularizer to loss wich force that layer to retain it's properties. Latter could be very non-trivial $\endgroup$
    – mirror2image
    Jun 11, 2019 at 5:34

2 Answers 2

Reset to default
2
$\begingroup$

Extending @mirror2image's comment, if you have a certain metric that allows you to measure how close the intended layer is to a low pass filter (something that compares its output with what a low pass filter would have produced, for example), the simplest way to achieve what you want would be to add a term in your loss function that calculates the value of this metric. This way, each time you do a training step, the network now is not only made to output the correct predictions but is also forced to do so while also keeping that specific layer's behavior as close to a low-pass filter as possible. This is the most common way of tweaking the behavior of neural networks and is often encountered in many research papers.

Improve this answer
$\endgroup$
0
$\begingroup$

the idea about training that it is allowed for weights (not layer as you wrote) "learn" values what they "want" from general network setting. But actualy i v thinked about that too, and to have weights represent as you want, you can first train a shorter network, while having those weights (1) as very last, so you have maximum control over them, after trained a lot append next layers to those and learning rate for weights (1) make much smaller as for new weights

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .