Forcing a neural network to be close to a previous model - Regularization through given model

I'm wondering, has anyone seen any paper where one trains a network but biases it to produce similar outputs to a given model (such as one given from expert opinion or it being a previously trained network).

Formally, I'm looking for a paper doing the following:

Let $$g:\mathbb{R}^d\rightarrow \mathbb{R}^D$$ be a model (not necessarily, but possibly, a neural network) trained on some input/output data pairs $$\{(x_n,y_n)\}_{n=1}^N$$ and train a neural network $$f_{\theta}(\cdot)$$ on $$\underset{\theta}{\operatorname{argmin}}\sum_{n=1}^N \left\| f_{\theta}(x_n) - y_n \right\| + \lambda \left\| f_{\theta}(x_n) - g(x_n) \right\|,$$ where $$\theta$$ represents all the trainable weight and bias parameters of the network $$f_{\theta}(\cdot)$$.

So put another way...$$f_{\theta}(\cdot)$$ is being regularized by the outputs of another model...

• This reminds me of the ELBO objective function, which is composed of two terms: one is the likelihood and the other the KL divergence between the variational density and the prior. Why does your problem/question remind me of the ELBO? Because, essentially, the prior here would be your previously trained model while the variational density is what you are looking. You might want to have a look at it for inspiration (at least). But what's the problem with your current approach of using some kind of norm to regularize? – nbro Jul 24 '20 at 1:07
• @nbro I'll have a look at the ELBO objective function; thanks I also had some type of pre-training idea in mind. To answer your question, basically I was able to show universality of certain models arising from this type of regularization so I'm looking to connect it to pre-existing literature. – BLBA Jul 24 '20 at 6:24
• Well, actually, this also reminds me of the GAN. See deepgenerativemodels.github.io/notes/gan. – nbro Jul 24 '20 at 19:51
• @nbro I gave this some more thought and is this related to transfer learning, where, the second term dictates how for a NN departs from a previously trained model? – BLBA Jul 27 '20 at 11:58
• No, I have not seen it, but I just wanted to point you the idea behind the GAN, where there are two models. Of course, you will have to change the objective function to do what you want. – nbro Jul 27 '20 at 13:57