Why is my loss (binary cross entropy) converging on ~0.6? (Task: Natural Language Inference)

Question

I’m trying to debug my neural network (BERT fine-tuning) trained for natural language inference with binary classification of either entailment or contradiction. I've trained it for 80 epochs and its converging on ~0.68. Why isn't it getting any lower?

Thanks in advance!

---

Neural Network Architecture:

Training details:

• Loss function: Binary cross entropy
• Batch size: 8
• Optimizer: Adam (learning rate = 0.001)
• Framework: Tensorflow 2.0.1
• Pooled embeddings used from BERT output.
• BERT parameters are not frozen.

Dataset:

• 10,000 samples
• balanced dataset (5k each for entailment and contradiction)
• dataset is a subset of data mined from wikipedia.
• Claim example: "'History of art includes architecture, dance, sculpture, music, painting, poetry literature, theatre, narrative, film, photography and graphic arts.'"
• Evidence example: "The subsequent expansion of the list of principal arts in the 20th century reached to nine : architecture , dance , sculpture , music , painting , poetry -LRB- described broadly as a form of literature with aesthetic purpose or function , which also includes the distinct genres of theatre and narrative -RRB- , film , photography and graphic arts ."

Dataset preprocessing:

• Used [SEP] to separate the two sentences instead of using separate embeddings via 2 BERT layers. (Hence, segment ids are computed as such)
• BERT's FullTokenizer for tokenization.
• Truncated to a maximum sequence length of 64.

See below for a graph of the training history. (Red = train_loss, Blue = val_loss)

Answer 1

It seems to be overfitting and your model is not learning. Try SGD optimizer with a learning rate of 0.001 ADAM optimizer will give you a soon overfitting, and decreasing the learning rate will train your model better. The learning rate is about steps to change weights, in this plot you see that the validation loss is not changing with an optimization goal

Answer 2

Are you using BinaryCrossEntropy through tensorflow? If so, check if you are using the logits argument. I am using from_logits=True .It is not similar to the original BinaryCrossEntropy loss.