2
$\begingroup$

I read an article about captioning videos and I want to use solution number 4 (extract features with a CNN, pass the sequence to a separate RNN) in my own project.

But for me, it seems really strange that in this method we use the Inception model without any retraining or something like that. Every project has different requirements and even if you use pretrained model instead of your own, you should do some training.

And I wonder how to do this? For example, I created a project where I use the network with CNN layers and then LSTM and Dense layers. And in every epoch, there is feed-forward and backpropagation through the whole network, all layers. But what if you have CNN network to extract features and LSTM network that takes sequences as inputs. How to train CNN network if there is no defined output? This network should only extract features but the network doesn't know what features. So the question is: How to train CNN to extract relevant features and then passing these features to LSTM?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

The approach that you don't train the whole net, but just the latter part of it (all starting with lstm in our case), can actually work. The idea is that the inception was already pretrained a very large dataset (imagenet for instance). And it's capable of extracting some useful information from it. Actually there are different domains of images in the imagenet and the inception net needed to capture a vast variety of input information to classify images well. The idea is that the pretrained inception is already capable to extract almost everything what could possibly be useful (unless your images aren't something completely different from imagenet, but that a rare case). Then you adapt the lstm layers and the fully connected layers to correctly process that information. Maybe you aren't going to get the perfect score with this approach and maybe it's better to train the whole large net including the inception part on the new data to lower the distributional shift and that's what people usually do in fact, but it takes more time to train and if you don't have enough data you won't be able to achieve results that are significantly better than those with a frozen CNN part.

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ Thank you for a good reply! But this creates another questions. You say that in order to have good results I need to train the whole network, not only the lstm part. But in this case I should create the model which takes sequence of images as input (for example 5 frames) and train this big model. It requires more data and time or gpu power to do this, but it's not a big problem. But there is another problem. $\endgroup$
    – user40943
    Commented Sep 12, 2020 at 19:52
  • $\begingroup$ It may be very slow while testing. Because you take 5 frames, you do CNN part, then LSTM part and you have the output. And then you capture new frame. And you must do the whole CNN part once again for 2,3,4,5 and 6 frame. So you do CNN calculations second time for 2,3,4 and 5 frame which is a terrible waste of time. But your model expects 5 images as input so you must do it. $\endgroup$
    – user40943
    Commented Sep 12, 2020 at 19:52
  • $\begingroup$ But if you have separate CNN to extract features, you can extract features for last 5 frames and then pass these features to RNN. And then you do CNN part for 6th frame and you pass the features from 2,3,4,5,6 frames to RNN which is better. The task I want to do is autonomous driving using sequences of images. I don't know if using pre-trained Inception or VGG network I will have good results, but I could try. But I am just curious how to do it from scratch. I may have to do this in the feature or maybe even in this project if pre-trained networks fail. $\endgroup$
    – user40943
    Commented Sep 12, 2020 at 19:52
  • $\begingroup$ Should I train CNN with one image as input and steering command as output and define LSTM network with a sequence of features as input and steering command as output. So in other words, should the output of the CNN and LSTM part be defined the same way? And then you train CNN on a lot of images, and at the end you train LSTM using the middle conv/pooling layer with features as inputs? $\endgroup$
    – user40943
    Commented Sep 12, 2020 at 19:52
  • $\begingroup$ No, it doesn't work like that. You can compute the CNN part frame-wise only once and then feed the outputs to the lstm just once. It's like you take a frame, compute output of the CNN, feed it into the lstm part and update its hidden and cell states. After that you take the second frame and do it all over again. That's the conventional way to work with sequential data. $\endgroup$
    – Michał Słodki
    Commented Sep 12, 2020 at 22:26
0
$\begingroup$

you could also just use a Task-agnostic CNN as an encoder to get extract features like in (1) and then use the output of the last global pooling layer and then feed that as an input to the LSTM layer or any other downstream task. Add another small Neural Network (projection head) after the CNN. And then use contrastive loss on output of this projection head to improve upon the model.

(1) Big Self-Supervised Models are Strong Semi-Supervised Learners (Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey Hinton ) Link: https://arxiv.org/abs/2006.10029

Improve this answer
$\endgroup$

You must log in to answer this question.