Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
I’m trying to debug my neural network (BERT fine-tuning) trained for natural language inference with binary classification of either entailment or contradiction. I've trained it for 80 epochs and its converging on ~0.68. Why isn't it getting any lower?
Thanks in advance!
Neural Network Architecture:
Loss function: Binary cross entropy
Batch size: 8
Optimizer: Adam (learning rate = 0.001)
Framework: Tensorflow 2.0.1
Pooled embeddings used from BERT output.
BERT parameters are not frozen.
balanced dataset (5k each for entailment and contradiction)
dataset is a subset of data mined from wikipedia.
Claim example: "'History of art includes architecture, dance, sculpture, music, painting, poetry literature, theatre, narrative, film, photography and graphic arts.'"
Evidence example: "The subsequent expansion of the list of principal arts in the 20th century reached to nine : architecture , dance , sculpture , music , painting , poetry -LRB- described broadly as a form of literature with aesthetic purpose or function , which also includes the distinct genres of theatre and narrative -RRB- , film , photography and graphic arts ."
Used [SEP] to separate the two sentences instead of using separate embeddings via 2 BERT layers. (Hence, segment ids are computed as such)
It seems to be overfitting and your model is not learning. Try SGD optimizer with a learning rate of 0.001
ADAM optimizer will give you a soon overfitting, and decreasing the learning rate will train your model better. The learning rate is about steps to change weights, in this plot you see that the validation loss is not changing with an optimization goal