I have created a simple XLMRoberta model for token classification. The task is to predict the quality of translation for each token/word.

The data looks something like this, where the first sentence is source and next is translated, which is combined input to my model. And the last two entities are my label that corresponds to OK or BAD for each token in source and target sentences.

NOTE that tokens in target(last entry of array list) are more because labels have to be predicted for each gap between 2 words as well to predict if one or more missing words should have been there.

array(['these passages also celebrate a curvaceous stomach and midriff and plumpness as aspects of female physical attractiveness .', 'diese Passagen feiern auch einen kurvenreichen Magen und midriff und Plumpness als Aspekte der weiblichen körperlichen Attraktivität .', 'OK BAD OK BAD BAD BAD BAD BAD BAD OK BAD BAD OK OK OK OK OK OK', 'OK OK OK OK OK BAD OK OK OK OK BAD BAD OK BAD OK OK OK BAD OK BAD OK BAD OK OK OK OK OK OK OK BAD OK OK OK OK BAD OK OK'], dtype=object)

I have almost 7k data for fine tuning the model. The model is learning nothing till 3 epochs, i am not sure what is the issue. I even tried augmentating the data with synonyms but no help.

Here's my training function :

def train_fn(data_loader, model, optimizer, scheduler):
total_train_loss = 0
lst_active_labels = []
lst_active_preds = []

for batch in tqdm(data_loader, total = len(data_loader)):
    b_input_ids = batch[0].cuda()
    b_input_mask = batch[1].cuda()
    b_labels = batch[2].cuda()
    # Zero the gradients
    outputs = model(b_input_ids, 

    loss = outputs[0]
    total_train_loss += loss.item()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

return total_train_loss / len(data_loader)

This the model :

class EntityModel(nn.Module):

def __init__(self):
    super(EntityModel, self).__init__()
    self.bert = XLMRobertaForTokenClassification.from_pretrained(config.BASE_MODEL,output_attentions = False, output_hidden_states = False, num_labels=2)

def forward(self, ids, attention_mask, labels):
    outputs = self.bert(ids,
                            attention_mask = attention_mask,
                            labels = labels,return_dict=False)

    return outputs[0], outputs[1]

Am I missing anything? Or it can be that the problem is such that its difficult for the language model to learn? Input contains two sentences : [SRC] SEP [TAR] SEP. Parameters i have tried already are : lr : 2e-5, 2e-4, 1e-3 with decay rate of 0.01, 0.001 and 0.001 respectively.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.