I'm training a language model with
5000 vocabularies using a single
M60 GPU (w/ actually usable memory about 7.5G).
The number of tokens per batch is about
8000, and the hidden dimension to the softmax layer is
512. So, if I understand correctly, fully-connected (softmax) layer theoretically consumes
5000*8000*512*4=81.92GB for a forward pass (4 is for float32).
But the GPU performed the forward and backward passes without any problem, and it says the GPU memory usage is less than
7GB in total.
I used PyTorch. What's causing this?
EDIT: To be clearer, the input to the final fc layer (256x5000 matrix) is of size [256, 32, 256].