I’m trying to debug my neural network (BERT fine-tuning) trained for natural language inference with binary classification of either entailment or contradiction. I've trained it for 80 epochs and its converging on ~0.68. Why isn't it getting any lower?
Thanks in advance!
Neural Network Architecture:
Training details:
- Loss function: Binary cross entropy
- Batch size: 8
- Optimizer: Adam (learning rate = 0.001)
- Framework: Tensorflow 2.0.1
- Pooled embeddings used from BERT output.
- BERT parameters are not frozen.
Dataset:
- 10,000 samples
- balanced dataset (5k each for entailment and contradiction)
- dataset is a subset of data mined from wikipedia.
- Claim example: "'History of art includes architecture, dance, sculpture, music, painting, poetry literature, theatre, narrative, film, photography and graphic arts.'"
- Evidence example: "The subsequent expansion of the list of principal arts in the 20th century reached to nine : architecture , dance , sculpture , music , painting , poetry -LRB- described broadly as a form of literature with aesthetic purpose or function , which also includes the distinct genres of theatre and narrative -RRB- , film , photography and graphic arts ."
Dataset preprocessing:
- Used [SEP] to separate the two sentences instead of using separate embeddings via 2 BERT layers. (Hence, segment ids are computed as such)
- BERT's FullTokenizer for tokenization.
- Truncated to a maximum sequence length of 64.
See below for a graph of the training history. (Red = train_loss, Blue = val_loss)