I have two models and a file contains captions for images. The output of model 1 is .pkl files that contain the features of the images. Model 2 is the language model that will be trained with the captions. How can I link between two models to predict a caption for any image? The output of model 1 should be the input of model 2. But the features only are not enough so the input of model 2 will be .pkl files + caption file. Right?

If someone can help me in getting the link between the two models, I will appreciate it.


1 Answer 1


The Standard Image Captioning Pipeline is to train the model in a single batch(or mini-batch) i.e. get the features from the CNN Image encoder and then feed that into an RNN decoder (features + Real Captions) to produce output captions for the Image.

The training loop in PyTorch would look something like this:

# zero the parameter gradients
# Forward pass
features = encoder(image)
outputs = decoder(features, captions)
# Compute the Loss
loss = criterion(outputs.view(-1, vocab_size), 
# Backward pass.
# Update the parameters in the optimizer.

I'd suggest you go through the paper Show and Tell: A Neural Image Caption Generator.

I also made this Kaggle Kernel implementing the paper from scratch. Should help clear up any other doubts.

  • $\begingroup$ thanks for replying , should I train the language model for the captions of my own dataset separately? Will I need the class code separately to link between the two models? $\endgroup$ Commented Mar 30, 2021 at 16:05
  • $\begingroup$ Yes you should train the RNN decoder for your particular dataset separately again (for best performance). I'm not able to understand your second question, can you please clarify $\endgroup$ Commented Mar 30, 2021 at 16:09
  • $\begingroup$ i have the two models separately .. i didn't write the code that makes a caption for the image yet. how can i start for the linking between them $\endgroup$ Commented Mar 30, 2021 at 16:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .