When it comes to using Transformers for image captioning is there any reason to use masking?

I currently have a resnet101 encoder and am trying to use the features as the input for a transformer model in order to generate a caption for the image, is there any need to use masking? and what would I mask if I did need to?

Any help would be much appreciated

Thanks in advance.


If you're using a library such as Trax which contains great submodules for various Transformers (Skipping, BERT, Vanilla and Reformer) you can use the inbuilt trax.data.inputs.add_loss_weights() function and provide a value for the id_to_mask parameter.

Example Usage:

train_generator = trax.data.inputs.add_loss_weights(
data_generator(batch_size, x_train, y_train,vocab['<PAD>'], True),

Here are some resources for building Transformers in Trax:

  • 1
    $\begingroup$ The associated blog post link gives me a 404. Is there an up to date version? $\endgroup$ Dec 29 '20 at 13:36
  • $\begingroup$ Here's the updated link. Sorry I moved my blog from fastpages 😅 $\endgroup$ Dec 29 '20 at 14:45
  • 1
    $\begingroup$ @SauravMaheshkar It seems the link is still not working. $\endgroup$ Apr 25 '21 at 19:59
  • $\begingroup$ I keep changing blog platforms a lot so the links keep changing. I'll make it a point to add a permalink next time. For now, the link is this. I apologize but the Markdown parser is still not fixed so the latex is going to be hard to read. If you necessarily need the latex rendered visit this alternative link $\endgroup$ Apr 26 '21 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.