# Confusion Matrix Measures vs Accuracy level in Neural Network Model

I'm working on a classification machine learning problem with two classes: high and low, which are derived from another numerical column x. Previously, if x>100, the sample is considered high, otherwise, it is considered low. I used a 1D CNN model.

I wanted to test if changing the threshold would impact on the model performance. So I increased the threshold to 1000(ie. if x>1000, the sample is considered high, otherwise, it is considered low. ).

What I did:

1. Increased the threshold from 100 to 1000, thus the number of samples that fall into the low category increased and that of the high category decreased. The data thus become imbalanced with the following ratio:
Ratio of low : high

(with 100 threshold)       43% : 57%

(with 1000 threshold)       8% : 92%

1. Noticing that the imbalanced dataset might lead to imbalanced training data, I applied undersampling to the training data to make sure that there are equal number of low and high samples.
2. At 100 threshold, the model achieved an accuracy level of around 80%. At 1000 threshold, the model achieved an accuracy level of around 69%, which is significantly lower than that of the 100 threshold.
3. I then saved each model and generated a prediction using each model. I evaluated both sets of results with the Precision, Recall and F1 Score.

Precision: 0.38
(with 100 threshold)   Recall: 0.86
F1: 0.527

Precision: 0.8307692307692308
(with 1000 threshold)  Recall: 0.6206896551724138
F1: 0.7105263157894737


Based on accuracy level, it seems 100 threshold achieved a better performance than that of 1000 threshold. However, according to F1 Score, 1000 threshold seems to achieve better score 0.71 (as compared to 0.527 of 100 threshold).

My question are:

1. Is it possible for a model to have lower accuracy level but higher F1 Score? How can this make sense in light of confusion-matrix measures and model accuracy level?
2. Which measures should I use to determine using which threshold is better?

Edit:

Since the entire code is too long, I will show some snippets of it below.

To answer @MASTER OF CODE's question, I believe the test data is balanced and tested in the validation stage at the end of each epoch as I called model.fit(x_train,y_train, validation) in Keras which used the built-in API for training and evaluation(please cmiiw).

For Undersampling:

I used np.random.choice() to randomly select an equal amount of majority samples as the minority group, then concatenated the minority and majority samples into one dataframe named under_sample, before fitting a 1D CNN model on the dataframe, see below:

np.random.seed(42)
############ minority_class_len
minority_class_len = len(df[df['Label'] ==1])

############ majority_class_indices
majority_class_indices = df[df['Label'] ==0].index

############ random_majority_indices
random_majority_indices = np.random.choice(majority_class_indices,
minority_class_len,
replace = False)

############ minority_class_indices
minority_class_indices = df[df['Label'] ==1].index

############ concatenate positive and negative sample indices
under_sample_indices = np.concatenate([minority_class_indices, random_majority_indices])

############ select samples by indices
under_sample = df.loc[under_sample_indices]
$$$$
`
• Is Your testing set also balanced? Also could You show some code to recreate this behaviour? Feb 3, 2022 at 10:02
• Hi @MASTEROFCODE, thanks for your reply, please see question edit. Unfortunately I'm unable to upload the entire code but I added some snippets, hopefully it makes it a bit clear. Feb 3, 2022 at 13:09

$$F1= \frac{2 TP}{2TP + FP + FN}$$ 