0
$\begingroup$

I have a question about the context of CNN and LSTM. I have trained a CNN network for image classification. However, I would like to combine it with LSTM for visualizing the attention weights. So, I extracted the features from the CNN to put it into LSTM. However, I am stuck at the concept of combinating the CNN with LSTM.

– Do I need to train the whole network again? Or just training the LSTM part is fine? – Can I just train the LSTM on image sequences based on classes (for e.g. 1 class has around 300 images) and do predictions later on extracted video frames? - In what way can I implement the attention mechanism with Keras?

I hope you can help me while I struggle with the context of understanding the combination of this.

~ EDITED ~

I have trained a resnet50 to classify images. Although, I removed the last dense layer, to extract features from the trained CNN network. Those extracted features will be used as input in the newly created LSTM with attention mechanism to find out where the focus lies. The predictions will be on videos (extracted frames).

Image -> extract features (CNN) -> LSTM + Attention (to check where the focus lies during the prediction) -> classify image (output class from N labels)

$\endgroup$
  • $\begingroup$ what is the time (or sequential) component of your system that you want an lstm for? (is it a video? like array of images?) $\endgroup$ – mshlis Jul 22 at 17:00
  • $\begingroup$ @mshlis Yes it's an array of images $\endgroup$ – Joker Jul 22 at 20:34
  • $\begingroup$ can you explain more about the CNN you trained and what you want to transfer over for what goal? $\endgroup$ – mshlis Jul 22 at 23:08
  • $\begingroup$ @mshlis updated the question. See ~ EDITED part ~. $\endgroup$ – Joker Jul 22 at 23:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.