I am working on a very basic binary classification problem. For each set of four float numbers $(x,y,z,w)$, I want to check if they fall or not into one category.
I have written a model with 3 dense layers (ReLU activation function) and an output layer (with sigmoid activation function). The model doesn't overfit, so I tried increasing the hyper-parameters, but still it doesn't overfit. I thought that reaching overfit was easy, as long as you increase the number of nodes. Isn't true?
Initially, I thought the problem was with the data, so I have decided to generate a mock dataset, but still the model doesn't overfit. In the code below, the function generate_pattern()
generate a valid pattern that I want to label with the integer 1. I populate a Pandas dataframe with this function, and I add some noise by inserting random generated patterns.
Why is the model not overfitting? What is the best model's architecture for this kind of problems?
import pandas as pd
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from sklearn.model_selection import train_test_split
def generate_pattern():
[x,y,z,w] = np.random.rand(4)/50
return [0.35+x,0.45+y,0.7+z,1.32+w]
mock_data = pd.DataFrame(columns=['x','y','z','w','target'])
i=0
while i < 10000:
if np.random.randint(2) == 0:
mock_data.loc[i] = generate_pattern() +[1]
i+=1
else:
if np.random.randint(2) == 0:
if np.random.randint(2) == 1:
mock_data.loc[i] = list(np.random.rand(4)) + [0]
i+=1
df_input = mock_data[['x','y','z','w']]
df_output = mock_data[['target']]
X = df_input.values
Y = df_output.astype(int).values
X_train, X_test, y_train, y_test = train_test_split(X, Y[:,0], test_size=0.33, random_state=52)
X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.33, random_state=52)
# Create the neural network model
model = Sequential()
model.add(Dense(5, activation='relu', input_dim=4))
model.add(Dense(16, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
# Compile the model
model.compile(loss=['binary_crossentropy'], optimizer='rmsprop', metrics=['accuracy','mean_squared_error','binary_crossentropy'])
# Train the model
history = model.fit(X_train, y_train, epochs=150, batch_size=64, validation_data=(X_valid,y_valid))