I want to use frames from video game and analyze them using CNN and LSTM. But when I have the model defined like that
frames, channels, rows, columns = 5,3,224,224 video = Input(shape=(frames, rows, columns, channels)) cnn_base = VGG16(input_shape=(rows, columns, channels), weights="imagenet", include_top=False) cnn_base.trainable = False cnn_out = GlobalAveragePooling2D()(cnn_base.output) cnn = Model(cnn_base.input, cnn_out) encoded_frames = TimeDistributed(cnn)(video) encoded_sequence = LSTM(256)(encoded_frames) hidden_layer = Dense(1024, activation="relu")(encoded_sequence) outputs = Dense(10, activation="softmax")(hidden_layer) model = Model(video, outputs) model.summary()
it is very slow. Because you must feed 5 frames to the model and it works like that:
You feed 1,2,3,4 and 5 frame Then you feed 2,3,4,5 and 6 frame Then you feed 3,4,5,6 and 7 frame
And every time the CNN calculations are done for every frame instead of only the new one. It's terrible waste of time. I think it should work that for every new frame there are CNN calculations and then you feed extracted features from previous frames (and the newest frame) to the LSTM. So only LSTM part must be done from the beginning (because it is the new sequence).
I think one solution would be to have two separate networks - CNN and LSTM. So for every frame you can calculate features using CNN that takes 1 image as input and then you feed features to LSTM. Is it ok? If yes, how to train such CNN to extract useful features? In order to train CNN to extract useful features you must define some output for this CNN (without defined output you cannot train any network). So how I should define output of the CNN? The same way as the output of the LSTM or CNN-LSTM network? I want to use later the last pooling layer from trained CNN to feed features from this layer to the LSTM. But the important question is: how to define output of the CNN network? The same way as the output of the LSTM network?
My task is autonomous driving but I think it doesn't matter here. The code is just an example. I want to train my own network from scratch, not use pretrained network like VGG.