2
$\begingroup$

I’m trying to debug my neural network (BERT fine-tuning) trained for natural language inference with binary classification of either entailment or contradiction. I've trained it for 80 epochs and its converging on ~0.68. Why isn't it getting any lower?

Thanks in advance!

Neural Network Architecture:

Architecture

Training details:

  • Loss function: Binary cross entropy
  • Batch size: 8
  • Optimizer: Adam (learning rate = 0.001)
  • Framework: Tensorflow 2.0.1
  • Pooled embeddings used from BERT output.
  • BERT parameters are not frozen.

Dataset:

  • 10,000 samples
  • balanced dataset (5k each for entailment and contradiction)
  • dataset is a subset of data mined from wikipedia.
  • Claim example: "'History of art includes architecture, dance, sculpture, music, painting, poetry literature, theatre, narrative, film, photography and graphic arts.'"
  • Evidence example: "The subsequent expansion of the list of principal arts in the 20th century reached to nine : architecture , dance , sculpture , music , painting , poetry -LRB- described broadly as a form of literature with aesthetic purpose or function , which also includes the distinct genres of theatre and narrative -RRB- , film , photography and graphic arts ."

Dataset preprocessing:

  • Used [SEP] to separate the two sentences instead of using separate embeddings via 2 BERT layers. (Hence, segment ids are computed as such)
  • BERT's FullTokenizer for tokenization.
  • Truncated to a maximum sequence length of 64.

See below for a graph of the training history. (Red = train_loss, Blue = val_loss)

0.68 convergence

Improve this question
$\endgroup$
1
  • $\begingroup$ I ran into a similar problem and my issue was that I had a Sigmoid layer at the end of the model (that I thought I had removed). So I had Sigmoid->BCEWithLogitsLoss, effectively doing two sigmoids after another. Funnily enough, it was working anyway when I was doing Sigmoid->Sigmoid->L1... so not sure how much of an answer this is. Worth double-checking if anyone has the same issue though. $\endgroup$
    – lucidbrot
    Jul 5 at 16:23

2 Answers 2

Reset to default
2
$\begingroup$

It seems to be overfitting and your model is not learning. Try SGD optimizer with a learning rate of 0.001 ADAM optimizer will give you a soon overfitting, and decreasing the learning rate will train your model better. The learning rate is about steps to change weights, in this plot you see that the validation loss is not changing with an optimization goal

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hi your method worked, thank you! I'm trying to get a better insight into why it is. I'm plotting the trainable parameters on TensorBoard, do you have any recommendations as to what I should look out for? Also I have a follow up post here. @SahaTib $\endgroup$
    – Jack-P
    Mar 8, 2020 at 10:51
  • $\begingroup$ @Jack-P glad to hear that, check this out:machinelearningmastery.com/… it may help $\endgroup$
    – SahaTib
    Mar 9, 2020 at 22:29
  • 1
    $\begingroup$ Thanks for the resource! I’m actually trying to understand why SGD was able to overfit the model (I’m following the general advice to first overfit a model to make sure it works) while Adam couldn’t as evident from the high training loss. Is it perhaps because it’s stuck at a saddle point or a local minima but the stochastic nature of SGD was able to escape? @SahaTib $\endgroup$
    – Jack-P
    Mar 10, 2020 at 7:57
0
$\begingroup$

Are you using BinaryCrossEntropy through tensorflow? If so, check if you are using the logits argument. I am using from_logits=True .It is not similar to the original BinaryCrossEntropy loss.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .