How to manage large amounts of image data for training?

Question

Right now, I am trying to synthesize training images for a CNN and due to the nature of the application, there is a finite number of sample images to learn from.

From other research, I expect to be using about 200,000 training images at a resolution of 1280*720, which with 3 channel at 8 bits will take about 550 GB to save uncompressed. This number can and probably will rise in the future, meaning more memory that I will need to provide.

I imagine that there are applications that required even more training data with higher complexity and that there are solutions to handling that such as compression techniques and the like.

My question: Are there solutions for the memory management beyond compressing the images with JPEG and such besides generating and instantly consuming the pictures without saving them to permanent memory?

Answer 1

I suggest you fine-tune an existing model. Knowledge transfer models in many image processing tasks are now open sourced and you can build your model on top of them. Also, knowledge transfer models are trained on a large datasets and can quickly converge to your case-study with a little of task-specific extra training.

This way you will use few data to tune the model which leads to less memory use and less training time. You will also take advantage from a ready-to-use architecture and get accurate results.

Depending on your case-study you can choose from this list of computer vision pre-trained models.

Answer 2

Most of the algorithms seem accessing the training set sequentially so the images need not be loaded into memory all at the same time.

It is technically fully possible to build a workstation with 1 Tb of RAM or more, using a server barebone in a tower form factor (see this, for instance, and would support multiple GPUs) but this only makes sense if the image loading is a bottleneck. Current SSDs are rather fast so you need to measure this before spending money on such a beast.