How can I reduce the GPU memory usage with large images?

Question

I am trying to train a CNN-LSTM model. The size of my images is 640x640. I have a GTX 1080 ti 11GB. I am using Keras with the TensorFlow backend.

Here is the model.

img_input_1 = Input(shape=(1, n_width, n_height, n_channels))
conv_1 = TimeDistributed(Conv2D(96, (11,11), activation='relu', padding='same'))(img_input_1)
pool_1 = TimeDistributed(MaxPooling2D((3,3)))(conv_1)
conv_2 = TimeDistributed(Conv2D(128, (11,11), activation='relu', padding='same'))(pool_1)
flat_1 = TimeDistributed(Flatten())(conv_2)
dense_1 = TimeDistributed(Dense(4096, activation='relu'))(flat_1)
drop_1 = TimeDistributed(Dropout(0.5))(dense_1)
lstm_1 = LSTM(17, activation='linear')(drop_1)
dense_2 = Dense(4096, activation='relu')(lstm_1)
dense_output_2 = Dense(1, activation='sigmoid')(dense_2)
model = Model(inputs=img_input_1, outputs=dense_output_2)

op = optimizers.Adam(lr=0.00001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.001)

model.compile(loss='mean_absolute_error', optimizer=op, metrics=['accuracy'])

model.fit(X, Y, epochs=3, batch_size=1)

Right now, using this model, I can only use the training data when the images are resized to 60x60, any larger and I run out of GPU memory.

I want to use the largest possible size as I want to retain as much discriminatory information as possible. (The labels will be mouse screen coordinates between 0 - 640).

Among many others, I found this question: How to handle images of large sizes in CNN?

Though I am not sure how I can "restrict your CNN" or "stream your data in each epoch" or if these would help.

How can I reduce the amount of memory used so I can increase the image sizes?

Is it possible to sacrifice training time/computation speed in favor of higher resolution data whilst retaining model effectiveness?

Note: the above model is not final, just a basic outlay.

Answer 1

As the other links suggest, you basically have four options:

• "restrict your CNN". This means making your model smaller and simpler, possibly by inserting a pooling layer at the front, or reducing the total number of layers. From a memory perspective, this isn't likely to produce really large gains though.
• "stream your data in each epoch". By default, the entire training set will be stored on the GPU. This is a good idea, because the bus connecting the GPU and the RAM has extremely high latency. It takes a very long time to start sending data across the bus. However, most systems have much more space in the RAM than in the video memory on a GPU. Instead of storing all the training data in the GPU, you could store it in main memory, and then manually move over just the batch of data you want to use for a given update. After computing the update, you could free the memory assigned to the batch. I am not sure how to do this in Keras. In the past, I have done this by writing a custom CUDA kernel.
• "Use less data". If you train on a random subset of the training data, you can keep your images at high quality, but your model will probably overfit. If you train on downsampled images, your model may not be able to discriminate between them well. However, both of these options are easy to do.
• "Get more memory". If you buy a video card with more video RAM, you can train more complex models. This is why scientific-grade cards have much larger memories than gaming cards. The Tesla V100 has 32GB of memory, and 16GB cards are common, whereas even the most advanced cards for gaming have only 11GB.