Ways to Reduce False Positive or False Negatives in Binary Classification (0,1)

Question

I am working on a task in which I need to classify binary labels 0 and 1 properly (as close to perfection as possible). My final dataset (ready for classification) has input data with 141 features and binary labels. The original data is a time-series data generated using inductive sensor and the features are extracted from it using Short Time Fourier Transform. Moreover, the dataset is highly imbalanced, total number of label 0 is 39263 whereas that of label 1 is 71 ( with 39334 total number of samples).

I have developed a Neural Network classification model using PyTorch. I have used the following chunk of code in order to split the data into Train, Test and Validation datasets.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=1 - train_ratio, stratify=y, shuffle=True)
X_val, X_test, y_val, y_test = train_test_split(X_test, y_test, test_size=test_ratio/(test_ratio + validation_ratio), stratify=y_test)

In order to handle class imbalance issue, I have generated class weights using sklearn tool, assigned the weights to each sample and then passed the weights as an argument to CrossEntropy Loss Function. Class weight for label 0 is 0.50090416 whereas that for label 1 is 277.

class_weights = class_weight.compute_class_weight(class_weight='balanced', classes=np.unique(y), y=y)
class_weights = torch.tensor(class_weights, dtype=torch.float)
sample_weights = sklearn.utils.class_weight.compute_sample_weight(class_weight= {0:0.50090416, 1:277}, y= y, indices=None)

Following is my model:

# number of features (len of X cols)
input_dim = 141
# number of hidden layers
hidden_layers = 5
# number of classes (unique of y)
output_dim = 2
class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.linear1 = nn.Linear(input_dim, hidden_layers)
        self.relu = nn.ReLU()
        self.linear2 = nn.Linear(hidden_layers, output_dim)
        self.drop_layer = nn.Dropout(p=0.25)

    def forward(self, x):
        x = self.drop_layer(x)
        hidden = self.linear1(x)
        relu = self.relu(hidden)
        x = self.drop_layer(x)
        output = self.linear2(relu)
        return output

clf = Network()

And the loss function and optimizer has been defined as follows:

criterion_weighted = nn.CrossEntropyLoss(weight=class_weights, reduction='mean')
optimizer = torch.optim.SGD(clf.parameters(), lr=0.001, momentum=0.9, weight_decay=0.01)

Training, validation and testing phases are represented by the following code snippet:

batch size = 20
epoch = 10
t_accuracy_gain = []
accuracy_gain = []
for epoch in range(epochs):

    # Training
    training_loss = 0
    total, total_t = 0, 0
    for train_input, train_labels in trainloader:
        # set optimizer to zero grad to remove previous epoch gradients
        optimizer.zero_grad()
        y_pred = clf(train_input)
        loss = criterion_weighted(y_pred, train_labels)
        loss.backward()
        # optimize
        optimizer.step()
        training_loss += loss.item()
        y_pred = torch.nn.functional.softmax(y_pred, dim=1)
        for i, p in enumerate(y_pred):
            if train_labels[i] == torch.max(p.data, 0)[1]:
                total_t = total_t+1

    accuracy_t = total_t/train_size
    t_accuracy_gain.append(accuracy_t)

    # Validating
    valid_loss = 0.0
    for val_inputs, val_labels in valloader:
        # Forward Pass
        y_pred_val = clf(val_inputs)
        # Find the Loss
        loss = criterion_weighted(y_pred_val, val_labels)
        # Calculate Loss
        valid_loss += loss.item()
        y_pred_val = torch.nn.functional.softmax(y_pred_val, dim=1)
        for i, p in enumerate(y_pred_val):
            if val_labels[i] == torch.max(p.data, 0)[1]:
                total = total+1

    accuracy = total/val_size
    accuracy_gain.append(accuracy)

    print(f'Epoch {epoch+1} \t\t Training Loss: { training_loss/len(trainloader)} \t\t Validation Loss: { valid_loss/len(valloader)}')
    print(accuracy_t, accuracy)

    epoch += 1

# Testing

test = []
with torch.no_grad():
    correct = 0
    for i, X in enumerate(X_test):
        y_pred = clf(X)
        if y_pred.argmax().item() == y_test[i]:
            correct += 1
        test.append(y_pred.argmax().item())
print(f'{correct} out of {y_test.shape[0]} is correct : {correct/y_test.shape[0]*100}%')
print(np.unique(test, return_counts=True))
print(test)
print(confusion_matrix(y_test, test))
print('Precision: %.16f' % precision_score(y_test, test))
print('Recall: %.16f' % recall_score(y_test, test))
print('F1_score: %.16f' % f1_score(y_test, test))

Now, the results I am obtaining are very frustrating. Although I am assigning weights to my classes, still the model is unable to classify the minority class. The following are the results obtained after 10 epochs:

Epoch 1          Training Loss: 0.18776317898373482          Validation Loss: 0.15920198694881746
0.9982033898305085 0.998135593220339
Epoch 2          Training Loss: 0.15280305894249577          Validation Loss: 0.15878191357952054
0.9982033898305085 0.998135593220339
Epoch 3          Training Loss: 0.14959193905152507          Validation Loss: 0.15635720060791
0.9982033898305085 0.998135593220339
Epoch 4          Training Loss: 0.14844961922426345          Validation Loss: 0.15760730873730222
0.9982033898305085 0.998135593220339
Epoch 5          Training Loss: 0.14479931982518252          Validation Loss: 0.15391030547729992
0.9982033898305085 0.998135593220339
Epoch 6          Training Loss: 0.14200111128010992          Validation Loss: 0.15406650864219262
0.9982033898305085 0.998135593220339
Epoch 7          Training Loss: 0.13820280621238684          Validation Loss: 0.15289437643931073
0.9982033898305085 0.998135593220339
Epoch 8          Training Loss: 0.13983591395540762          Validation Loss: 0.15828328180616186
0.9982033898305085 0.998135593220339
Epoch 9          Training Loss: 0.13601937021113047          Validation Loss: 0.15396189160518728
0.9982033898305085 0.998135593220339
Epoch 10         Training Loss: 0.1354785456634679       Validation Loss: 0.15328112466860624
0.9982033898305085 0.998135593220339
3927 out of 3934 is correct : 99.8220640569395%

As can be observed from the above metrics, the model is classifying all the 0 labels properly, but 1 labels have been incorrectly classified as false negatives.

After this, I randomly oversampled the training and validation dataset using SMOTE technique and the following are the results that I have obtained:

Epoch 1          Training Loss: 0.19352763494530437          Validation Loss: 1.5280835092738145
0.9684348150915204 0.5
Epoch 2          Training Loss: 0.26620812748063905          Validation Loss: 1.5480152127951232
0.928753353482528 0.5
Epoch 3          Training Loss: 0.2296104698174701       Validation Loss: 1.5715507436132592
0.9466159540870037 0.5
Epoch 4          Training Loss: 0.21165923436025255          Validation Loss: 1.585304197290808
0.9547661901042551 0.5
Epoch 5          Training Loss: 0.1981869825486138       Validation Loss: 1.596927644055596
0.9594186164974361 0.5
Epoch 6          Training Loss: 0.19129694016564935          Validation Loss: 1.6035434622475673
0.9616769110605494 0.5
Epoch 7          Training Loss: 0.188139278968521        Validation Loss: 1.6061198817904412
0.9625428736373824 0.5
Epoch 8          Training Loss: 0.18814285206584758          Validation Loss: 1.605698257425392
0.9628145481712908 0.5
Epoch 9          Training Loss: 0.18444137671465885          Validation Loss: 1.6085442355272623
0.9639521852820321 0.5
Epoch 10         Training Loss: 0.181928029220001        Validation Loss: 1.6101362825085128
0.9645294936665875 0.5
7 out of 3934 is correct : 0.1779359430604982%

In this scenario, the model is able to classify all the 1s perfectly but is unable to classify 0s. And this should also be noticed that the validation loss is much higher than the training loss which may mean that the model is overfitting in the training phase.

Without sampling, the model is predicting the minority class as false negatives, whereas with sampling, the model is predicting the majority class as false positives. As a newbie in this field, I fail to understand how to resolve this error. Can anybody help?

Answer 1

First, it is slightly odd that you have an output dimension of 2, when you are doing binary classification. In binary classification, it is logical to have one output node, where 0 implies one class and 1 implies another class.

Then to add to that. If you have one node as output, you can throw an activation function over that (like you do right now with the softmax function). You then obtain an output between 0 and 1. Normally, you would round off an output to get the prediction (0.45 -> label 0, 0.51 -> label 1). However, if you would like to balance what your model predicts, you could set the threshold differently, for example, on 0.7. This would lead to your model predicting more samples as 0 than as 1.

To have better insights into what changing such a threshold does to your output, you could compute an ROC curve for all thresholds you would want to test out.