I want to pre-train a model (combined by two popular modules A and B, and both are large blocks), then fine-tune it on downstream tasks.
What if for the weight initialization for pre-training, module A is initialized from some checkpoints, while B's is from random? Can I still claim the process as pre-training? Or the modules in the model must all be initialized from random, and so we can call it pre-training?
If parts of the module's weights are 'contaminated' by checkpoints, it is only can be called fine-tune?