3
$\begingroup$

Let's say one wants to use a neural net to learn some function $g(x)$. Let's say that we know that $g$ is a combination of two functions (or two sub-problems), $g(x)=f_2(f_1(x))$, and that we have two datasets

  1. composed of $x$ samples and their corresponding $g(x)$ labels, and
  2. composed of $x$ samples and their corresponding $f_1(x)$ labels.

Should we use two nets, one to learn the mapping from $x$ samples to $f_1(x)$ using dataset 1 and another net to learn the mapping from $f_1(x)$ to $g(x)$ (note that we can build a dataset composed of $f_1(x)$ samples and $g(x)$ labels with the trained net), or just one net to learn mappings from $x$ to $g(x)$ using dataset 1?

Intuitively, the first option seems to be better since we take advantage of our knowledge that $f_1$ is a "sub-problem" of $g$.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The tendency in literature in the last years (at least for computer vision problems) seems to point towards the single model option (I'll try to remember to come back and add some links to papers mentioning this when I find them), although this IMO is really data- and problem-dependent.

In your case, I would set up a network for the mapping $x$ to $g(x)$, with a training-only auxiliary loss calculated on the mapping $x$ to $f1(x)$ and compare this with a model trained only on "$x$ to $g(x)$".

Improve this answer
$\endgroup$
2
  • $\begingroup$ I didn't understand your second paragraph. Did you mean to say that you'd try both options and see which one is better? $\endgroup$
    – Gilad Deutsch
    Apr 29, 2020 at 11:24
  • 1
    $\begingroup$ No. I'd try the second option and compare it with a model that predicts both F1(X) and g(X). The model has two regression heads, one "earlier" in the network for F1(X) and one deeper that regresses g(X). You train as a multi-task problem using a combined weighed loss on both regression targets, but then for inference you only use the output from the head that predicts g(X). The idea is to use the first regression head to steer the training in the "easier" direction by incorporating the knowledge that predicting F1(X) helps predicting G(X). $\endgroup$
    – GPhilo
    Apr 29, 2020 at 11:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .