I am able to train 345M parameter GPT-2 using DialoGPT on Reddit data topics like AskReddit, Askmen, AskWomen, casualConvo, etc. And I using AWS p3dn.24xlarge instance with 256GB GPU. It is trained successfully.
But now I want to train a GPT-2 model on 1.5 billion parameters with 38GB of preprocessed data of Reddit, Blogspot, and other news sites. It is not getting trained on p3dn.24xlarge instance single instance.
So, how much amount of GPU do I need to train a DialoGPT2 with 1.5 billion parameters? Are there any techniques to train on Low 256GB GPU using some optimization technique?