I am in the process of collecting a huge dataset of Human poses captured images to create a model to classify poses.

My question is how will I be able to train on this massive dataset? I have multiple GPUs and Multiple machines access (Also have GCP).

What would be the best way to train on such huge dataset?


  • $\begingroup$ Welcome to Ai Community. Please read and follow the posting guidelines in the help documentation, as suggested when you created this account. Minimal, complete, verifiable example applies here. We cannot effectively help you until you post a sneak peak of what you have tried out and accurately specify the problem.Sometimes,we leave out questions unanswered due to lack of a brief but descriptive summary of the problem as the title of the question might say. $\endgroup$
    – quintumnia
    Aug 22 '19 at 2:52

is your data stored in raw ASCII text, like a CSV file?

Perhaps you can speed up data loading and use less memory by using another data format. A good example is a binary format like GRIB, NetCDF, or HDF.

There are many command line tools that you can use to transform one data format into another that do not require the entire dataset to be loaded into memory.

Using another format may allow you to store the data in a more compact form that saves memory, such as 2-byte integers, or 4-byte floats.

In some cases, you may need to resort to a big data platform. That is, a platform designed for handling very large datasets, that allows you to use data transforms and machine learning algorithms on top of it.

  • $\begingroup$ My data is in CSV having two columns, img_path and label. Image_path points to an actual image on which I have to train my CNN Model. So loading millions of images in memory is the main issue. $\endgroup$ Aug 26 '19 at 10:56
  • $\begingroup$ how did u label your images ? did u use any tool or software ?. $\endgroup$ Aug 26 '19 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.