1
$\begingroup$

I'm training a language model with 5000 vocabularies using a single M60 GPU (w/ actually usable memory about 7.5G).
The number of tokens per batch is about 8000, and the hidden dimension to the softmax layer is 512. So, if I understand correctly, fully-connected (softmax) layer theoretically consumes 5000*8000*512*4=81.92GB for a forward pass (4 is for float32).
But the GPU performed the forward and backward passes without any problem, and it says the GPU memory usage is less than 7GB in total.

I used PyTorch. What's causing this?

EDIT: To be clearer, the input to the final fc layer (256x5000 matrix) is of size [256, 32, 256].

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ You should clearly mention the what does the values stand for.. Because last time one person had a similar problem but his memory consumption was more as compared to less. $\endgroup$
    – user9947
    Commented Sep 17, 2018 at 8:53

1 Answer 1

Reset to default
1
$\begingroup$

GPU DRAM capacity - 7.5G the below link explains how nVIDIA GPU cUDNN does memory optimization. https://devblogs.nvidia.com/optimizing-recurrent-neural-networks-cudnn-5/ the below link has detailed steps to calculate memory required by parameters and data. http://cs231n.github.io/convolutional-networks/#case one data point missing in question is number of output classes in softmax layer. The above two links will help you to calculate memory required and how software handles large matrix multiplication.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you. The number of output classes is 5000, which is the number of vocabulary. Apparently, I didn't need to multiply 5000, so the memory consumption is at the order of 10MB. $\endgroup$
    – Math.StackExchange
    Commented Sep 17, 2018 at 20:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .