0
$\begingroup$

To assess our deep learning models (CNN) we have labeled a big benchmark dataset (it was labeled by specialists so it is kind of Ideal). I of course know that we do not want to train new models using the benchmark dataset since it would be "cheating ourselves". The question is if we can train a small "secret model" based only on the benchmark, and use it only for faster labeling of other "real" datasets that unlike the benchmark dataset WILL be used to train our models. Does such an approach sound legitimate?

$\endgroup$

2 Answers 2

1
$\begingroup$

neural network are function approximators...

What you are saying is that given a data distribution $D$, you train a model $f$ on it, and then you you se $f$ to label a new dataset $D'$

Now, you want to train a new model $g$ on $D'$... however, $D'$ is labeled by $f$, thus $g$ will approximate the function learned by $f$, which is supposedly a good function for $D$, thus $g$ will be close to $f$, thus it's pretty much saying that $g$ will be trained on $D$


I'm rephrasing with all the definitions:

  • $S$ is the specialist
  • $D$ the small dataset (benchmark)
  • $f$ is a neural network trained on $D$
  • $D'$ is a second dataset labeled by $f$
  • $g$ is a neural network trained on $D'$

the labels of $D'$ are point-wise evaluations of $f$, therefore when you will ask to $g$ to approximate them, you will have a new function ($g$) that approximate $f$

Now, depending on how related is $D'$ to $D$, you will have more or less "leakage" from $f$ to $g$

Conclusion: this method is effective only if $D'$ is similar to $D$, but if that's the case, then you will have a lot of leakage from $f$ to $g$, and thus you will have a final function $g$ that is almost trained on $D$

$\endgroup$
6
  • $\begingroup$ first, let's settle the terms: S - specialists that label the D, D = benchmark dataset, f= small secret model trained on benchmark dataset, D' = real dataset labeled by the f, g = real model trained on D ---- Did I get you right? (S is my addition for the upcoming question) If yes: The vanilla approach is that S (specialists) labels both D and D' But in an ideal world the S and the f would label D' similarly. So what is the difference? if you are right it should be prohibited for S (specialists) label both D and D') $\endgroup$
    – Igor
    Jan 29 at 11:35
  • $\begingroup$ @Igor updated the answer $\endgroup$
    – Alberto
    Jan 29 at 12:57
  • $\begingroup$ Thank you. But still, if you take an alternative scenario: the example where the same S(specialist) labels both - D and D', and in both cases does it ideally, you will again get the same leakage, won't you? $\endgroup$
    – Igor
    Jan 31 at 6:51
  • $\begingroup$ @Igor i mean, not really, but at that point, why do you need a small benchmark dataset, when you have a big one already labeled by an expert $\endgroup$
    – Alberto
    Jan 31 at 13:03
  • $\begingroup$ maybe there is a misunderstanding. There are 2 scenarios: 1) The specialist labels only the benchmark, and I based on it train a "secret" model that is being used INSTEAD of the specialist to label the "real-dataset" to train a "- real model". End of scenario 1. 2 (standard pipeline)I use the specialist to label both: the benchmark (as in the previous scenario) and the real dataset (which in the previous scenario was labeled by a model substituting the specialist but a very good-ideal model ideally substituting him) Would you say that both cases are the same in terms of leakage amount? $\endgroup$
    – Igor
    Jan 31 at 14:58
1
$\begingroup$

I'm not sure I understand the point. The secret model will invariably have some loss. This loss will be overlaid on the larger dataset as the secret model operates on it. Whatever model is trained from this larger dataset will learn the labels as given including the loss. If we then assume this final model also has a loss it likely learns a lossy version of the secret model's loss. Which theoretically could be a more accurate version of the secret model but more than likely will be less.

If I was forced to steel man this idea, given a known confusion one could use the secret model to label the larger dataset and focus specialists on correcting common confusion points. This should be faster than labeling the whole dataset. But without this step it's an invariably lossy model intimating an invariably lossy model.

$\endgroup$
1
  • $\begingroup$ The question wasn't whether the procedure would work better than directly training on the benchmark test set but whether it is cheating. $\endgroup$ Mar 9 at 19:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .