0
$\begingroup$

Say that we're on GPT20 - maybe the model that's resulted from training is 10PB large (maybe unlikely but this is an example). Is it possible for a GPT model to be distributed across machines? How does this work if distributed inference is needed?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
2
$\begingroup$

ChatGPT is already a model of Large neural networks based on a distributed computing platform. ChatGPT was built in collaboration with Microsoft. The model is trained using Azure supercomputing infrastructure with Nvidia GPUs. ChatGPT’s hardware comprises over 285,000 CPU cores, 10,000 GPUs, and network connectivity of 400 GBs per second per GPU server. Information taken from here and here

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .