Confusion Matrix Measures vs Accuracy level in Neural Network Model

Question

I'm working on a classification machine learning problem with two classes: high and low, which are derived from another numerical column x. Previously, if x>100, the sample is considered high, otherwise, it is considered low. I used a 1D CNN model.

I wanted to test if changing the threshold would impact on the model performance. So I increased the threshold to 1000(ie. if x>1000, the sample is considered high, otherwise, it is considered low. ).

What I did:

1. Increased the threshold from 100 to 1000, thus the number of samples that fall into the low category increased and that of the high category decreased. The data thus become imbalanced with the following ratio:

Ratio of low : high

(with 100 threshold)       43% : 57%

(with 1000 threshold)       8% : 92%

2. Noticing that the imbalanced dataset might lead to imbalanced training data, I applied undersampling to the training data to make sure that there are equal number of low and high samples.
3. At 100 threshold, the model achieved an accuracy level of around 80%. At 1000 threshold, the model achieved an accuracy level of around 69%, which is significantly lower than that of the 100 threshold.
4. I then saved each model and generated a prediction using each model. I evaluated both sets of results with the Precision, Recall and F1 Score.

                       Precision: 0.38
(with 100 threshold)   Recall: 0.86
                       F1: 0.527


                       Precision: 0.8307692307692308 
(with 1000 threshold)  Recall: 0.6206896551724138 
                       F1: 0.7105263157894737

Based on accuracy level, it seems 100 threshold achieved a better performance than that of 1000 threshold. However, according to F1 Score, 1000 threshold seems to achieve better score 0.71 (as compared to 0.527 of 100 threshold).

My question are:

1. Is it possible for a model to have lower accuracy level but higher F1 Score? How can this make sense in light of confusion-matrix measures and model accuracy level?
2. Which measures should I use to determine using which threshold is better?

---

Edit:

Since the entire code is too long, I will show some snippets of it below.

To answer @MASTER OF CODE's question, I believe the test data is balanced and tested in the validation stage at the end of each epoch as I called model.fit(x_train,y_train, validation) in Keras which used the built-in API for training and evaluation(please cmiiw).

For Undersampling:

I used np.random.choice() to randomly select an equal amount of majority samples as the minority group, then concatenated the minority and majority samples into one dataframe named under_sample, before fitting a 1D CNN model on the dataframe, see below:

np.random.seed(42)
############ minority_class_len
minority_class_len = len(df[df['Label'] ==1])


############ majority_class_indices
majority_class_indices = df[df['Label'] ==0].index


############ random_majority_indices
random_majority_indices = np.random.choice(majority_class_indices,
                                           minority_class_len,
                                           replace = False)

############ minority_class_indices
minority_class_indices = df[df['Label'] ==1].index


############ concatenate positive and negative sample indices
under_sample_indices = np.concatenate([minority_class_indices, random_majority_indices])


############ select samples by indices
under_sample = df.loc[under_sample_indices]
```

Answer 1

1. Yes, accuracy can be lower than the F score for a set of predictions and targets. You can easily test it by generating a fixed array of predictions and a varying array of labels (or vice versa). You'll get a graph like in the image below. Notice that you could obtain a horizontally mirrored version of the same graph by inverting the labels (or predictions).

To understand why you need to keep in mid that the F score is a skewed metric,because in its calculation we ignore an entire class, i.e. the true negatives.

$$F1= \frac{2 TP}{2TP + FP + FN} $$

So if I have an unbalanced dataset with only 10 instances of class A and 90 of class B I'll get an accuracy of 90% by predicting only B, with an f score of 0%, but if I invert the classes (i.e. predict only A) I'll get an accuracy of 10% and an F score of 18%.

2. The safer answer is always: check all metrics. Every metric tells you a different story about the model performance, so it's always a good practice to not rely only on one. But it might depends also on your specific use case. Why do you need to classify between low and high (whatever low and high refer to)? Do you care about a specific class? Then the recall is the most interesting metric to keep an eye on. In general, your problem is similar to probability thresholding when calculating the ROC-AUC score so you might want to check it out and see if it's applicable also to dataset splitting (a bit impractical if a single training session of your model is long though), if it's applicable, it's surely more reliable than the F score.