# XLMRoberta loss remains constant over iterations for TokenClassification task

I have created a simple XLMRoberta model for token classification. The task is to predict the quality of translation for each token/word.

The data looks something like this, where the first sentence is source and next is translated, which is combined input to my model. And the last two entities are my label that corresponds to OK or BAD for each token in source and target sentences.

NOTE that tokens in target(last entry of array list) are more because labels have to be predicted for each gap between 2 words as well to predict if one or more missing words should have been there.

array(['these passages also celebrate a curvaceous stomach and midriff and plumpness as aspects of female physical attractiveness .', 'diese Passagen feiern auch einen kurvenreichen Magen und midriff und Plumpness als Aspekte der weiblichen körperlichen Attraktivität .', 'OK BAD OK BAD BAD BAD BAD BAD BAD OK BAD BAD OK OK OK OK OK OK', 'OK OK OK OK OK BAD OK OK OK OK BAD BAD OK BAD OK OK OK BAD OK BAD OK BAD OK OK OK OK OK OK OK BAD OK OK OK OK BAD OK OK'], dtype=object)

I have almost 7k data for fine tuning the model. The model is learning nothing till 3 epochs, i am not sure what is the issue. I even tried augmentating the data with synonyms but no help.

Here's my training function :

def train_fn(data_loader, model, optimizer, scheduler):
model.train()
total_train_loss = 0
lst_active_labels = []
lst_active_preds = []

b_input_ids = batch[0].cuda()
b_labels = batch[2].cuda()

outputs = model(b_input_ids,
labels=b_labels)

loss = outputs[0]

loss.backward()

total_train_loss += loss.item()

optimizer.step()
scheduler.step()



This the model :

class EntityModel(nn.Module):

def __init__(self):

super(EntityModel, self).__init__()
self.bert = XLMRobertaForTokenClassification.from_pretrained(config.BASE_MODEL,output_attentions = False, output_hidden_states = False, num_labels=2)