On-Policy Algorithms like PPO directly maximize the performance objective or an approximation of it. They tend to be quite stable and reliable but are often sample inefficient. Off-Policy Algorithms like TD3 improve the sample inefficiency by reusing data collected with previous policies, but they tend to be less stable. (Source: Kinds of RL Algorithms - Spinning up - OpenAI)
Looking at learning curves comparing SOTA algorithms, we see that off-policy algorithms quickly improve performance at the training's beginning. Here an example:
Can we start training off-policy and after some time use the learned and quickly improved policy to init the policy network of an on-policy algorithm?