I am working on a very basic binary classification problem. For each set of four float numbers $(x,y,z,w)$, I want to check if they fall or not into one category.
I have written a model with 3 dense layers (ReLU activation function) and an output layer (with sigmoid activation function). The model doesn't overfit, so I tried increasing the hyper-parameters, but still it doesn't overfit. I thought that reaching overfit was easy, as long as you increase the number of nodes. Isn't true?
Initially, I thought the problem was with the data, so I have decided to generate a mock dataset, but still the model doesn't overfit. In the code below, the function
generate_pattern() generate a valid pattern that I want to label with the integer 1. I populate a Pandas dataframe with this function, and I add some noise by inserting random generated patterns.
Why is the model not overfitting? What is the best model's architecture for this kind of problems?
import pandas as pd import numpy as np from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense from sklearn.model_selection import train_test_split def generate_pattern(): [x,y,z,w] = np.random.rand(4)/50 return [0.35+x,0.45+y,0.7+z,1.32+w] mock_data = pd.DataFrame(columns=['x','y','z','w','target']) i=0 while i < 10000: if np.random.randint(2) == 0: mock_data.loc[i] = generate_pattern() + i+=1 else: if np.random.randint(2) == 0: if np.random.randint(2) == 1: mock_data.loc[i] = list(np.random.rand(4)) +  i+=1 df_input = mock_data[['x','y','z','w']] df_output = mock_data[['target']] X = df_input.values Y = df_output.astype(int).values X_train, X_test, y_train, y_test = train_test_split(X, Y[:,0], test_size=0.33, random_state=52) X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.33, random_state=52) # Create the neural network model model = Sequential() model.add(Dense(5, activation='relu', input_dim=4)) model.add(Dense(16, activation='relu')) model.add(Dense(32, activation='relu')) model.add(Dense(1, activation='sigmoid')) # Compile the model model.compile(loss=['binary_crossentropy'], optimizer='rmsprop', metrics=['accuracy','mean_squared_error','binary_crossentropy']) # Train the model history = model.fit(X_train, y_train, epochs=150, batch_size=64, validation_data=(X_valid,y_valid))