I am working on a task in which I need to classify binary labels 0 and 1 properly (as close to perfection as possible). My final dataset (ready for classification) has input data with 141 features and binary labels. The original data is a time-series data generated using inductive sensor and the features are extracted from it using Short Time Fourier Transform. Moreover, the dataset is highly imbalanced, total number of label 0 is 39263 whereas that of label 1 is 71 ( with 39334 total number of samples).
I have developed a Neural Network classification model using PyTorch. I have used the following chunk of code in order to split the data into Train, Test and Validation datasets.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=1 - train_ratio, stratify=y, shuffle=True)
X_val, X_test, y_val, y_test = train_test_split(X_test, y_test, test_size=test_ratio/(test_ratio + validation_ratio), stratify=y_test)
In order to handle class imbalance issue, I have generated class weights using sklearn tool, assigned the weights to each sample and then passed the weights as an argument to CrossEntropy Loss Function. Class weight for label 0 is 0.50090416 whereas that for label 1 is 277.
class_weights = class_weight.compute_class_weight(class_weight='balanced', classes=np.unique(y), y=y)
class_weights = torch.tensor(class_weights, dtype=torch.float)
sample_weights = sklearn.utils.class_weight.compute_sample_weight(class_weight= {0:0.50090416, 1:277}, y= y, indices=None)
Following is my model:
# number of features (len of X cols)
input_dim = 141
# number of hidden layers
hidden_layers = 5
# number of classes (unique of y)
output_dim = 2
class Network(nn.Module):
def __init__(self):
super(Network, self).__init__()
self.linear1 = nn.Linear(input_dim, hidden_layers)
self.relu = nn.ReLU()
self.linear2 = nn.Linear(hidden_layers, output_dim)
self.drop_layer = nn.Dropout(p=0.25)
def forward(self, x):
x = self.drop_layer(x)
hidden = self.linear1(x)
relu = self.relu(hidden)
x = self.drop_layer(x)
output = self.linear2(relu)
return output
clf = Network()
And the loss function and optimizer has been defined as follows:
criterion_weighted = nn.CrossEntropyLoss(weight=class_weights, reduction='mean')
optimizer = torch.optim.SGD(clf.parameters(), lr=0.001, momentum=0.9, weight_decay=0.01)
Training, validation and testing phases are represented by the following code snippet:
batch size = 20
epoch = 10
t_accuracy_gain = []
accuracy_gain = []
for epoch in range(epochs):
# Training
training_loss = 0
total, total_t = 0, 0
for train_input, train_labels in trainloader:
# set optimizer to zero grad to remove previous epoch gradients
optimizer.zero_grad()
y_pred = clf(train_input)
loss = criterion_weighted(y_pred, train_labels)
loss.backward()
# optimize
optimizer.step()
training_loss += loss.item()
y_pred = torch.nn.functional.softmax(y_pred, dim=1)
for i, p in enumerate(y_pred):
if train_labels[i] == torch.max(p.data, 0)[1]:
total_t = total_t+1
accuracy_t = total_t/train_size
t_accuracy_gain.append(accuracy_t)
# Validating
valid_loss = 0.0
for val_inputs, val_labels in valloader:
# Forward Pass
y_pred_val = clf(val_inputs)
# Find the Loss
loss = criterion_weighted(y_pred_val, val_labels)
# Calculate Loss
valid_loss += loss.item()
y_pred_val = torch.nn.functional.softmax(y_pred_val, dim=1)
for i, p in enumerate(y_pred_val):
if val_labels[i] == torch.max(p.data, 0)[1]:
total = total+1
accuracy = total/val_size
accuracy_gain.append(accuracy)
print(f'Epoch {epoch+1} \t\t Training Loss: { training_loss/len(trainloader)} \t\t Validation Loss: { valid_loss/len(valloader)}')
print(accuracy_t, accuracy)
epoch += 1
# Testing
test = []
with torch.no_grad():
correct = 0
for i, X in enumerate(X_test):
y_pred = clf(X)
if y_pred.argmax().item() == y_test[i]:
correct += 1
test.append(y_pred.argmax().item())
print(f'{correct} out of {y_test.shape[0]} is correct : {correct/y_test.shape[0]*100}%')
print(np.unique(test, return_counts=True))
print(test)
print(confusion_matrix(y_test, test))
print('Precision: %.16f' % precision_score(y_test, test))
print('Recall: %.16f' % recall_score(y_test, test))
print('F1_score: %.16f' % f1_score(y_test, test))
Now, the results I am obtaining are very frustrating. Although I am assigning weights to my classes, still the model is unable to classify the minority class. The following are the results obtained after 10 epochs:
Epoch 1 Training Loss: 0.18776317898373482 Validation Loss: 0.15920198694881746
0.9982033898305085 0.998135593220339
Epoch 2 Training Loss: 0.15280305894249577 Validation Loss: 0.15878191357952054
0.9982033898305085 0.998135593220339
Epoch 3 Training Loss: 0.14959193905152507 Validation Loss: 0.15635720060791
0.9982033898305085 0.998135593220339
Epoch 4 Training Loss: 0.14844961922426345 Validation Loss: 0.15760730873730222
0.9982033898305085 0.998135593220339
Epoch 5 Training Loss: 0.14479931982518252 Validation Loss: 0.15391030547729992
0.9982033898305085 0.998135593220339
Epoch 6 Training Loss: 0.14200111128010992 Validation Loss: 0.15406650864219262
0.9982033898305085 0.998135593220339
Epoch 7 Training Loss: 0.13820280621238684 Validation Loss: 0.15289437643931073
0.9982033898305085 0.998135593220339
Epoch 8 Training Loss: 0.13983591395540762 Validation Loss: 0.15828328180616186
0.9982033898305085 0.998135593220339
Epoch 9 Training Loss: 0.13601937021113047 Validation Loss: 0.15396189160518728
0.9982033898305085 0.998135593220339
Epoch 10 Training Loss: 0.1354785456634679 Validation Loss: 0.15328112466860624
0.9982033898305085 0.998135593220339
3927 out of 3934 is correct : 99.8220640569395%
As can be observed from the above metrics, the model is classifying all the 0 labels properly, but 1 labels have been incorrectly classified as false negatives.
After this, I randomly oversampled the training and validation dataset using SMOTE technique and the following are the results that I have obtained:
Epoch 1 Training Loss: 0.19352763494530437 Validation Loss: 1.5280835092738145
0.9684348150915204 0.5
Epoch 2 Training Loss: 0.26620812748063905 Validation Loss: 1.5480152127951232
0.928753353482528 0.5
Epoch 3 Training Loss: 0.2296104698174701 Validation Loss: 1.5715507436132592
0.9466159540870037 0.5
Epoch 4 Training Loss: 0.21165923436025255 Validation Loss: 1.585304197290808
0.9547661901042551 0.5
Epoch 5 Training Loss: 0.1981869825486138 Validation Loss: 1.596927644055596
0.9594186164974361 0.5
Epoch 6 Training Loss: 0.19129694016564935 Validation Loss: 1.6035434622475673
0.9616769110605494 0.5
Epoch 7 Training Loss: 0.188139278968521 Validation Loss: 1.6061198817904412
0.9625428736373824 0.5
Epoch 8 Training Loss: 0.18814285206584758 Validation Loss: 1.605698257425392
0.9628145481712908 0.5
Epoch 9 Training Loss: 0.18444137671465885 Validation Loss: 1.6085442355272623
0.9639521852820321 0.5
Epoch 10 Training Loss: 0.181928029220001 Validation Loss: 1.6101362825085128
0.9645294936665875 0.5
7 out of 3934 is correct : 0.1779359430604982%
In this scenario, the model is able to classify all the 1s perfectly but is unable to classify 0s. And this should also be noticed that the validation loss is much higher than the training loss which may mean that the model is overfitting in the training phase.
Without sampling, the model is predicting the minority class as false negatives, whereas with sampling, the model is predicting the majority class as false positives. As a newbie in this field, I fail to understand how to resolve this error. Can anybody help?
