I'm reading a program that use the pre-trained Roberta model (roberta-base). The code first extracts word embeddings from each caption in the batch, using the last hidden state of the Roberta model. Then, the model is trained to align these word embeddings with the image features (pixels) of the image through a type of attention mechanism. Then the models are updated using attention loss function. This iterative process continues until the training is complete, so I guess the word embeddings will be different after each epoch ? This is a multi-modal problem.

When I compare the Roberta model after training with the pre-trained model (roberta-base), I notice that every parameters the trained Roberta model are different, seems like the new model has updated the parameters. I'm not sure whether this is a form of fine-tuning or just feature extraction or both ?


1 Answer 1


According to this reference:

feature extraction involves creating new features that still capture the essential information from the original data but in a more efficient way.

Feature extraction techniques vary depending on the domain: Image Processing: Techniques like edge detection filters, Gabor filters, and Histogram of Oriented Gradients (HOG) can be used to extract features from images. Text Data: Natural Language Processing uses techniques such as Bag of Words, TF-IDF, and word embeddings to extract features from text.

Therefor your using the last hidden state of the Roberta model as new contextualized embedding feature of the input sequence for your downstream fine tuning task is feature extraction, and your whole approach contains both.

  • $\begingroup$ The contextualized embeddings in the first epoch are feature extraction because it takes directly from the pre-trained Roberta, but the contextualized embeddings for rest of epochs are different because the model has updated ? $\endgroup$
    – user77925
    Mar 8 at 22:22
  • $\begingroup$ Yes, as the fine-tuning process progresses through subsequent epochs, your model's parameters including those of RoBERTa and any additional layers added for the multimodal task, are updated based on the training data. As a result, the contextualized embeddings produced by RoBERTa in subsequent epochs may differ from those obtained in the initial epoch, reflecting the updated knowledge and representations learned by the model during training. $\endgroup$
    – cinch
    Mar 8 at 22:29
  • $\begingroup$ Thanks for your help $\endgroup$
    – user77925
    Mar 8 at 22:40

You must log in to answer this question.