0
$\begingroup$

I trained a 1D CNN model to model bacterial plate count based on time series data of water temperature. Bacterial place count is numerical, based on which I created two category variables, namely "Label 1" and "Label 2". There are two possible values for "Label 1" - "risky" and "non-risky" based on a threshold of 100 for bacterial place count . Eg. 1 if Bacterial place count >100; 0 otherwise. "Label 2" is (1 - value of "Label 2"). This gives a balanced "Label 1" data of 43.1% for 1 and 56.9% for 0. The model observed an 80% accuracy for prediction.

Then I wanted to see if changing the threshold would change the performance of the model. So I increased the threshold to 500 and trained the model, The ratio of 1 and 0 for "Label 1" is now 12.5% for 1 and 87.5% for 0. And at 1000 threshold, the ratio is 8.1% for 1 and 91.9% for 0.

Then I calculated the confusion matrix results for the predictions at each threshold, as below:

Confusion matrix using 100 threshold:

enter image description here

Confusion matrix using 500 threshold:

enter image description here

Then I further increased the threshold to 1000:

enter image description here

Calculated Results:

**Precision**
# 100 threshold
151/(151+246) = 38%

# 500 threshold
85/(85+30) = 74%

# 1000 threshold
47/(47+12) = 79%

**Recall**
# 100 threshold
151/(151+23) = 86%

# 500 threshold
85/(85+159) = 35%

# 1000 threshold
47/(47+127) = 27%

**F1-score**
# 100 threshold
2*(0.38*0.86)/(0.38+0.86) # 52.7%

# 500 threshold
2*(0.35*0.74)/(0.35+0.74) # 47.5%

# 1000 threshold
2*(0.79*0.27)/(0.79+0.27) # 40.2%

Something I observed:

  1. during training, there were consecutive training batches getting the same validation accuracy, for instance:

83/83 [==============================] - 1s 11ms/step - loss: 0.4065 - acc: 0.8347 - val_loss: 0.4527 - val_acc: 0.8152 Epoch 81/120 83/83 [==============================] - 1s 10ms/step - loss: 0.3943 - acc: 0.8552 - val_loss: 0.4350 - val_acc: 0.8152 Epoch 82/120 83/83 [==============================] - 1s 10ms/step - loss: 0.3800 - acc: 0.8408 - val_loss: 0.4601 - val_acc: 0.8152 Epoch 83/120 83/83 [==============================] - 1s 11ms/step - loss: 0.4026 - acc: 0.8396 - val_loss: 0.4351 - val_acc: 0.8152

I wonder if the model stops to learn as the data is highly imbalanced and so that the model would achieve higher accuracy simply by always predict the value with the higher weight. Is this a sign of the model freezing/not learning?

The graph looks like this: enter image description here

And my main questions are, does the numerical threshold used to define the classes able to have a significant impact on the accuracy level of the model? How do we interpret the confusion matrix and choose the model which gives the best performance? What are the metrics we should be considering? Any contribution is much appreciated.

Improve this question
$\endgroup$
2
  • $\begingroup$ Not sure I understand your first question (plus, you should really not ask multiple questions per post); isn't it obvious from your own experiments that the choice of the threshold can indeed have a (very) significant effect on the performance metrics? $\endgroup$
    – desertnaut
    Commented Jan 21, 2022 at 20:56
  • $\begingroup$ Hi @desertnaut thanks for the reply. Indeed the confusion matrix results are different. What I don't understand is if one threshold is necessarily better than the others. If yes, what are the criteria to select the best and the different consequences. My understanding is that precision and recall represent a trade-off between quality and quantity, since "Precision can be seen as a measure of quality, and recall as a measure of quantity". Also found this link but I'm not sure how exactly all this applies to my case. $\endgroup$
    – nilsinelabore
    Commented Jan 22, 2022 at 1:49

0

Reset to default

You must log in to answer this question.