I read an article about captioning videos and I want to use solution number 4 (extract features with a CNN, pass the sequence to a separate RNN) in my own project.
But for me, it seems really strange that in this method we use the Inception model without any retraining or something like that. Every project has different requirements and even if you use pretrained model instead of your own, you should do some training.
And I wonder how to do this? For example, I created a project where I use the network with CNN layers and then LSTM and Dense layers. And in every epoch, there is feed-forward and backpropagation through the whole network, all layers. But what if you have CNN network to extract features and LSTM network that takes sequences as inputs. How to train CNN network if there is no defined output? This network should only extract features but the network doesn't know what features. So the question is: How to train CNN to extract relevant features and then passing these features to LSTM?