I’m trying to debug my neural network (BERT fine-tuning) trained for natural language inference with binary classification of either entailment or contradiction. I've trained it for 80 epochs and its converging on ~0.68. Why isn't it getting any lower?

Thanks in advance!

Neural Network Architecture:


Training details:

  • Loss function: Binary cross entropy
  • Batch size: 8
  • Optimizer: Adam (learning rate = 0.001)
  • Framework: Tensorflow 2.0.1
  • Pooled embeddings used from BERT output.
  • BERT parameters are not frozen.


  • 10,000 samples
  • balanced dataset (5k each for entailment and contradiction)
  • dataset is a subset of data mined from wikipedia.
  • Claim example: "'History of art includes architecture, dance, sculpture, music, painting, poetry literature, theatre, narrative, film, photography and graphic arts.'"
  • Evidence example: "The subsequent expansion of the list of principal arts in the 20th century reached to nine : architecture , dance , sculpture , music , painting , poetry -LRB- described broadly as a form of literature with aesthetic purpose or function , which also includes the distinct genres of theatre and narrative -RRB- , film , photography and graphic arts ."

Dataset preprocessing:

  • Used [SEP] to separate the two sentences instead of using separate embeddings via 2 BERT layers. (Hence, segment ids are computed as such)
  • BERT's FullTokenizer for tokenization.
  • Truncated to a maximum sequence length of 64.

See below for a graph of the training history. (Red = train_loss, Blue = val_loss)

0.68 convergence


It seems to be overfitting and your model is not learning. Try SGD optimizer with a learning rate of 0.001 ADAM optimizer will give you a soon overfitting, and decreasing the learning rate will train your model better. The learning rate is about steps to change weights, in this plot you see that the validation loss is not changing with an optimization goal

  • $\begingroup$ Hi your method worked, thank you! I'm trying to get a better insight into why it is. I'm plotting the trainable parameters on TensorBoard, do you have any recommendations as to what I should look out for? Also I have a follow up post here. @SahaTib $\endgroup$ – Jack-P Mar 8 '20 at 10:51
  • $\begingroup$ @Jack-P glad to hear that, check this out:machinelearningmastery.com/… it may help $\endgroup$ – SahaTib Mar 9 '20 at 22:29
  • $\begingroup$ Thanks for the resource! I’m actually trying to understand why SGD was able to overfit the model (I’m following the general advice to first overfit a model to make sure it works) while Adam couldn’t as evident from the high training loss. Is it perhaps because it’s stuck at a saddle point or a local minima but the stochastic nature of SGD was able to escape? @SahaTib $\endgroup$ – Jack-P Mar 10 '20 at 7:57

Are you using BinaryCrossEntropy through tensorflow? If so, check if you are using the logits argument. I am using from_logits=True .It is not similar to the original BinaryCrossEntropy loss.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.