1
$\begingroup$

Problem (Sketch):

I'm interested in a particular formulation of the transfer-learning problem, which, given a trained network $f$ seeks to learn a new network $g$ whose last few layers behave very similarly to $f$. Thus, the "knowledge of f" is encoded into the output layers of $g$, but only approximately.

This is essentially what is described in the "fine-tuning" section of this Keras page. However, I'm looking for an academic reference..

Question Does anyone know of some article which use this kind of setup?

Formal Question:

Suppose that we have a feed-forward network $$ f(x):=f_n\circ \dots \circ f_1(x) \mbox{ where } f_i(x)=\sigma(A_ix +b_i),\, f_n(x)=A_n x+ b_n , $$ for $i=1,\dots,n-1$, trained on some given learning task. Given some $\epsilon>0$, and some loss functions $L_1,L_2$, I'm interested in transferring the "knowledge" of the network $f$ to a different learning task by training a new network $$ g(x):=g_N\circ \dots \circ g_1(x) \mbox{ where } g_i(x)=\sigma(A_i'x +b_i'),\, g_n(x)=A_n' x+ b_n; , $$ for $i=1,\dots,N-1$; by optimizing the following $$\begin{aligned} \min & \, L_2(g) \\ \mbox{ such that: } & L_1(f,g_N\circ\dots \circ g_{N-2} ) <\epsilon \end{aligned} $$

Related Post Here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.