Problem (Sketch):

I'm interested in a particular formulation of the transfer-learning problem, which, given a trained network $f$ seeks to learn a new network $g$ whose last few layers behave very similarly to $f$. Thus, the "knowledge of f" is encoded into the output layers of $g$, but only approximately.

This is essentially what is described in the "fine-tuning" section of this Keras page. However, I'm looking for an academic reference..

Question Does anyone know of some article which use this kind of setup?

Formal Question:

Suppose that we have a feed-forward network $$ f(x):=f_n\circ \dots \circ f_1(x) \mbox{ where } f_i(x)=\sigma(A_ix +b_i),\, f_n(x)=A_n x+ b_n , $$ for $i=1,\dots,n-1$, trained on some given learning task. Given some $\epsilon>0$, and some loss functions $L_1,L_2$, I'm interested in transferring the "knowledge" of the network $f$ to a different learning task by training a new network $$ g(x):=g_N\circ \dots \circ g_1(x) \mbox{ where } g_i(x)=\sigma(A_i'x +b_i'),\, g_n(x)=A_n' x+ b_n; , $$ for $i=1,\dots,N-1$; by optimizing the following $$\begin{aligned} \min & \, L_2(g) \\ \mbox{ such that: } & L_1(f,g_N\circ\dots \circ g_{N-2} ) <\epsilon \end{aligned} $$

Related Post Here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.