Right now, I am trying to synthesize training images for a CNN and due to the nature of the application, there is a finite number of sample images to learn from.
From other research, I expect to be using about 200,000 training images at a resolution of 1280*720, which with 3 channel at 8 bits will take about 550 GB to save uncompressed. This number can and probably will rise in the future, meaning more memory that I will need to provide.
I imagine that there are applications that required even more training data with higher complexity and that there are solutions to handling that such as compression techniques and the like.
My question: Are there solutions for the memory management beyond compressing the images with JPEG and such besides generating and instantly consuming the pictures without saving them to permanent memory?