Patterns binary classification - model doesn't overfit

Question

I am working on a very basic binary classification problem. For each set of four float numbers $(x,y,z,w)$, I want to check if they fall or not into one category.

I have written a model with 3 dense layers (ReLU activation function) and an output layer (with sigmoid activation function). The model doesn't overfit, so I tried increasing the hyper-parameters, but still it doesn't overfit. I thought that reaching overfit was easy, as long as you increase the number of nodes. Isn't true?

Initially, I thought the problem was with the data, so I have decided to generate a mock dataset, but still the model doesn't overfit. In the code below, the function generate_pattern() generate a valid pattern that I want to label with the integer 1. I populate a Pandas dataframe with this function, and I add some noise by inserting random generated patterns.

Why is the model not overfitting? What is the best model's architecture for this kind of problems?

import pandas as pd
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from sklearn.model_selection import train_test_split

def generate_pattern():
    [x,y,z,w] = np.random.rand(4)/50
    return [0.35+x,0.45+y,0.7+z,1.32+w]

mock_data = pd.DataFrame(columns=['x','y','z','w','target'])
i=0
while i < 10000:
    if np.random.randint(2) == 0:
        mock_data.loc[i] = generate_pattern() +[1]
        i+=1
    else:
        if np.random.randint(2) == 0:
            if np.random.randint(2) == 1:
                mock_data.loc[i] = list(np.random.rand(4)) + [0]
                i+=1

df_input = mock_data[['x','y','z','w']]
df_output = mock_data[['target']]
X = df_input.values
Y = df_output.astype(int).values
X_train, X_test, y_train, y_test = train_test_split(X, Y[:,0], test_size=0.33, random_state=52)
X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.33, random_state=52)


# Create the neural network model
model = Sequential()
model.add(Dense(5, activation='relu', input_dim=4))
model.add(Dense(16, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

# Compile the model
model.compile(loss=['binary_crossentropy'], optimizer='rmsprop', metrics=['accuracy','mean_squared_error','binary_crossentropy'])

# Train the model
history = model.fit(X_train, y_train, epochs=150, batch_size=64, validation_data=(X_valid,y_valid))

Answer 1

I think that the issue is still in the way you generate the fake data.

Try to look at your generated features as follows:

import seaborn as sns

sns.jointplot(data=mock_data, x='x', y='y', hue='target')
# or
sns.jointplot(data=mock_data, x='z', y='w', hue='target')

The second plot yields:

that basically means that if your model learns the rule (w > 1.1) and (z > 0.6) it can achieve perfect accuracy whatever test data you use. This should imply that the problem is too simple, and so overfitting is very unlikely.