0
$\begingroup$

I'm working on a simple MLP classification problem where I need to have the output layer as softmax.

The dataset that I've been using needs to pass through a parse that I made to remove NaN and change some text type labels to numeric type (attack = 1; natural = 0). Although, when I ran it, the accuracy, f1-score and recall are always outputting the same result and that's because of TP = TN and FP = FN every time.

The dataset is composed of 128 features and +1 marker label that indicates if it is an attack or a natural event, it's a very unbalanced dataset, ~70% attacks and ~30% naturals. The dataset can be found here Dataset, I'm using the "2-classes".

This is the parse function:

def NormalizeData(data):
    # print(f'min={np.min(data)} max={np.max(data)} mean={np.mean(data)}')

    if (np.max(data) == np.min(data)):
        return data
    
    return (data - np.min(data)) / (np.max(data) - np.min(data))

def parseDataset():
    path = '/content/gdrive/MyDrive/Colab Notebooks/binaryAllNaturalPlusNormalVsAttacks/'
    # all_files = glob.glob(os.path.join(path, "*.csv"))
    # dataset = pd.concat((pd.read_csv(f) for f in all_files))

    dataset = pd.read_csv(path + 'data1.csv');

    dataset = dataset.replace([np.inf, -np.inf], np.nan)
    dataset = dataset.fillna(dataset.mean()
    
    dataset = dataset.replace(['Attack'], 1) 
    dataset = dataset.replace(['Natural'], 0)

    # dt_attacks = dataset[dataset.marker == 1]
    # dt_naturals = dataset[dataset.marker == 0]
 
    csv_x = dataset.iloc[:, :-1].values
    csv_y = dataset.iloc[:, 128].values
    
    for i in range(0,128): # 0~127 -> 128 features
        csv_x[:,i] = NormalizeData(csv_x[:,i])

    return csv_x, csv_y

And the MLP:

# Configuration options
feature_vector_length = 128
num_classes = 2

# Load the data
x, y = parseDataset()

y = to_categorical(y, 2) # [[1. 0.] .... [0. 1.]]

train_x, test_x, train_y, test_y = train_test_split(x, y, test_size=0.30)

train_x = train_x.astype('float32')
test_x = test_x.astype('float32')

#print dataset
print('train_x',train_x.shape, train_x)
print('train_y', train_y.shape, train_y)

print('test_x', test_x.shape, test_x)
print('test_y', test_y.shape, test_y)

input_shape = (128,)

# Create the model
model = Sequential()
model.add(Dense(128, input_shape=input_shape, activation='relu'))
model.add(Dense(28, activation='relu'))
model.add(Dense(10, activation='relu'))
# model.add(Dense(2))
# model.add(keras.layers.Activation(tf.nn.softmax))
model.add(Dense(2, activation='softmax'))

# Configure the model and start training
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=[
                                                   'categorical_accuracy',
                                                    Precision(),
                                                    keras.metrics.Recall(),
                                                    keras.metrics.TruePositives(),
                                                    keras.metrics.TrueNegatives(),
                                                    keras.metrics.FalsePositives(),
                                                    keras.metrics.FalseNegatives()
                                                                ])

hist = model.fit(train_x,
                 train_y,
                 epochs=10,
                 batch_size=100)  

# Test the model after training
test_results = model.evaluate(test_x,
                              test_y, verbose=0)

print(f'Test results - Loss: {test_results[0]}')
print(f'Accuracy: {test_results[1]}')
print(f'Precision: {test_results[2]}')
print(f'Recall: {test_results[3]}')

Even with this being an unbalanced dataset, I would expect a good accuracy of ~70% but a bad precision and recall and big loss. But I don't really understand why it is happening with this dataset

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.