I have a question about the context of CNN and LSTM. I have trained a CNN network for image classification. However, I would like to combine it with LSTM for visualizing the attention weights. So, I extracted the features from the CNN to put it into LSTM. However, I am stuck at the concept of combinating the CNN with LSTM.
– Do I need to train the whole network again? Or just training the LSTM part is fine? – Can I just train the LSTM on image sequences based on classes (for e.g. 1 class has around 300 images) and do predictions later on extracted video frames? - In what way can I implement the attention mechanism with Keras?
I hope you can help me while I struggle with the context of understanding the combination of this.
~ EDITED ~
I have trained a resnet50 to classify images. Although, I removed the last dense layer, to extract features from the trained CNN network. Those extracted features will be used as input in the newly created LSTM with attention mechanism to find out where the focus lies. The predictions will be on videos (extracted frames).
Image -> extract features (CNN) -> LSTM + Attention (to check where the focus lies during the prediction) -> classify image (output class from N labels)
