0
$\begingroup$

I have two models and a file contains captions for images. The output of model 1 is .pkl files that contain the features of the images. Model 2 is the language model that will be trained with the captions. How can I link between two models to predict a caption for any image? The output of model 1 should be the input of model 2. But the features only are not enough so the input of model 2 will be .pkl files + caption file. Right?

If someone can help me in getting the link between the two models, I will appreciate it.

$\endgroup$
1
$\begingroup$

The Standard Image Captioning Pipeline is to train the model in a single batch(or mini-batch) i.e. get the features from the CNN Image encoder and then feed that into an RNN decoder (features + Real Captions) to produce output captions for the Image.

The training loop in PyTorch would look something like this:

# zero the parameter gradients
decoder.zero_grad()
encoder.zero_grad()
        
# Forward pass
features = encoder(image)
outputs = decoder(features, captions)
        
# Compute the Loss
loss = criterion(outputs.view(-1, vocab_size), 
                         captions.view(-1))
        
# Backward pass.
loss.backward()
        
# Update the parameters in the optimizer.
optimizer.step()

I'd suggest you go through the paper Show and Tell: A Neural Image Caption Generator.

I also made this Kaggle Kernel implementing the paper from scratch. Should help clear up any other doubts.

$\endgroup$
3
  • $\begingroup$ thanks for replying , should I train the language model for the captions of my own dataset separately? Will I need the class code separately to link between the two models? $\endgroup$ Mar 30 at 16:05
  • $\begingroup$ Yes you should train the RNN decoder for your particular dataset separately again (for best performance). I'm not able to understand your second question, can you please clarify $\endgroup$ Mar 30 at 16:09
  • $\begingroup$ i have the two models separately .. i didn't write the code that makes a caption for the image yet. how can i start for the linking between them $\endgroup$ Mar 30 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.