I'm wondering, has anyone seen any paper where one trains a network but biases it to produce similar outputs to a given model (such as one given from expert opinion or it being a previously trained network).

Formally, I'm looking for a paper doing the following:

Let $g:\mathbb{R}^d\rightarrow \mathbb{R}^D$ be a model (not necessarily, but possibly, a neural network) trained on some input/output data pairs $\{(x_n,y_n)\}_{n=1}^N$ and train a neural network $f_{\theta}(\cdot)$ on $$ \underset{\theta}{\operatorname{argmin}}\sum_{n=1}^N \left\| f_{\theta}(x_n) - y_n \right\| + \lambda \left\| f_{\theta}(x_n) - g(x_n) \right\|, $$ where $\theta$ represents all the trainable weight and bias parameters of the network $f_{\theta}(\cdot)$.

So put another way...$f_{\theta}(\cdot)$ is being regularized by the outputs of another model...

  • 1
    $\begingroup$ This reminds me of the ELBO objective function, which is composed of two terms: one is the likelihood and the other the KL divergence between the variational density and the prior. Why does your problem/question remind me of the ELBO? Because, essentially, the prior here would be your previously trained model while the variational density is what you are looking. You might want to have a look at it for inspiration (at least). But what's the problem with your current approach of using some kind of norm to regularize? $\endgroup$
    – nbro
    Jul 24, 2020 at 1:07
  • $\begingroup$ @nbro I'll have a look at the ELBO objective function; thanks I also had some type of pre-training idea in mind. To answer your question, basically I was able to show universality of certain models arising from this type of regularization so I'm looking to connect it to pre-existing literature. $\endgroup$
    – AIM_BLB
    Jul 24, 2020 at 6:24
  • $\begingroup$ Well, actually, this also reminds me of the GAN. See deepgenerativemodels.github.io/notes/gan. $\endgroup$
    – nbro
    Jul 24, 2020 at 19:51
  • $\begingroup$ @nbro I gave this some more thought and is this related to transfer learning, where, the second term dictates how for a NN departs from a previously trained model? $\endgroup$
    – AIM_BLB
    Jul 27, 2020 at 11:58
  • 1
    $\begingroup$ No, I have not seen it, but I just wanted to point you the idea behind the GAN, where there are two models. Of course, you will have to change the objective function to do what you want. $\endgroup$
    – nbro
    Jul 27, 2020 at 13:57


You must log in to answer this question.

Browse other questions tagged .