Problem (Sketch):
I'm interested in a particular formulation of the transfer-learning problem, which, given a trained network $f$ seeks to learn a new network $g$ whose last few layers behave very similarly to $f$. Thus, the "knowledge of f" is encoded into the output layers of $g$, but only approximately.
This is essentially what is described in the "fine-tuning" section of this Keras page. However, I'm looking for an academic reference..
Question Does anyone know of some article which use this kind of setup?
Formal Question:
Suppose that we have a feed-forward network $$ f(x):=f_n\circ \dots \circ f_1(x) \mbox{ where } f_i(x)=\sigma(A_ix +b_i),\, f_n(x)=A_n x+ b_n , $$ for $i=1,\dots,n-1$, trained on some given learning task. Given some $\epsilon>0$, and some loss functions $L_1,L_2$, I'm interested in transferring the "knowledge" of the network $f$ to a different learning task by training a new network $$ g(x):=g_N\circ \dots \circ g_1(x) \mbox{ where } g_i(x)=\sigma(A_i'x +b_i'),\, g_n(x)=A_n' x+ b_n; , $$ for $i=1,\dots,N-1$; by optimizing the following $$\begin{aligned} \min & \, L_2(g) \\ \mbox{ such that: } & L_1(f,g_N\circ\dots \circ g_{N-2} ) <\epsilon \end{aligned} $$
Related Post Here.
