Is it ok to pre-train and train (fine-tune) the neural network on the same training data?

Here is the specific context:

I am using the TabNet model on a tabular dataset. The dataset is fully labeled. I read about the performance and optimization benefits of pre-training the network. Can I pre-train it using the same training dataset on which I would eventually train my network? Or one is not supposed to re-use the data that was used for pre-training.

If I had to guess, I would say pre-training will use my training data to learn the dependencies between the features, which will help the training model in the supervised learning portion. I'm not sure if this is statistically sound or correct.

Is what I am about to do also called self-supervised learning?


1 Answer 1


As long as the first training is unsupervised and the second one supervised it does fall into the category of self-supervised learning.

I would say pre-training will use my training data to learn the dependencies between the features

This sentence is quite vague so let's clarify it a bit. Supervised and unsupervised losses are much different from each other statistically speaking. Supervised losses learn to predict conditional probabilities $p(y|x)$, i.e. the best predictions given an input. Unsupervised losses have no labels to compute such probabilities so they try to learn $p(x)$ directly, i.e. they try to learn the prior distribution of you data.

Of course between the two tasks the latter is much harder. Usually you would leverage labels to compute some distance or difference between predictions and labels but since you have no labels the only thing that can be done is to compute distances or differences between predictions and the same input that was fed to the model. So what an unsupervised model learn is not per se dependencies or correlations of features, is more generic than that, it's about learning to return similar predictions, usually embedding vectors, for similar inputs. Some approaches like contrastive learning try to go beyond that by imposing a margin between different predictions, so learning not only embedding vectors similar for similar inputs, but embedding vectors that are also dissimilar for dissimilar inputs.

This key aspect of unsupervised learning is the reason why it can be useful sometimes for pretraining supervised models. Supervised losses (almost all) are not designed to learn good embeddings. Problems like shift invariance and miss calibration are examples of consequences of that.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .