How can Image Caption work?

Question

I have two models and a file contains captions for images. The output of model 1 is .pkl files that contain the features of the images. Model 2 is the language model that will be trained with the captions. How can I link between two models to predict a caption for any image? The output of model 1 should be the input of model 2. But the features only are not enough so the input of model 2 will be .pkl files + caption file. Right?

If someone can help me in getting the link between the two models, I will appreciate it.

Answer 1

The Standard Image Captioning Pipeline is to train the model in a single batch(or mini-batch) i.e. get the features from the CNN Image encoder and then feed that into an RNN decoder (features + Real Captions) to produce output captions for the Image.

The training loop in PyTorch would look something like this:

# zero the parameter gradients
decoder.zero_grad()
encoder.zero_grad()
        
# Forward pass
features = encoder(image)
outputs = decoder(features, captions)
        
# Compute the Loss
loss = criterion(outputs.view(-1, vocab_size), 
                         captions.view(-1))
        
# Backward pass.
loss.backward()
        
# Update the parameters in the optimizer.
optimizer.step()

I'd suggest you go through the paper Show and Tell: A Neural Image Caption Generator.

I also made this Kaggle Kernel implementing the paper from scratch. Should help clear up any other doubts.