Gradient training changes indiscriminately all the weights and nodes of the neural network. But one can imagine the situations when the training should be shaped, e.g.:
- One can put constraints on some of the weights. E.g. human brain contains regions whose inner connections are more dense that external connections with different regions. One can try to mimic this region-shaped structure in neural networks as well and hence one can require that inter-regional weights (in opposit to intra-regional weights) are close to zero (except, possibly, for some channels among regions);
- One can put constraints on some of the weights in such manner that some layer of neurons have specific structure. E.g. consider the popular encoder-decoder architecture of neural machine translation e.g. https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html We can see that that the whole output of the encoder is expressed as a single layer of neurons which is forwarded to the input of the decoder. So - one can require that the set of all the possible outputs of the encoder (e.g. the possible values of this single layer of the neurons) forms some kind of structure, e.g. some grammar of some interlingua. This example is for illustration only, I have in mind more complex neural network which has one layer of neurons which indeed should output the encoded words of some interlingua. So, one is required to guid all the weights of the encoder in such manner that this single layer has only allowable values.
So - my question is - are there methods that guide the gradient descent training with additional information about the weights or about the nodes (i.e. about the whole subsets of weights that have some impact on specific layer of nodes)? E.g. about methods that impress the region structure on the neural network or that constrains the values of some nodes to be in specific range only?
Of course, it is quite easy to include such constraints in evolutionary neural networks - one can simply reject the neural networks with weights that violates the constraints. But is it possible to do this in gradient-like training?
