I have this binary regression model that has ~500 futures with an unbalanced dataset with the following results.

Loss is 0.06152007728815079,
Accuracy is 97.71724343299866

How I split the data:

X_train, X_val_and_test, Y_train, Y_val_and_test = train_test_split(X, Y, test_size=0.20, random_state=42)
X_validation, X_holdout_final_validation, Y_validation, Y_holdout_final_validation = train_test_split(
    X_val_and_test,
    Y_val_and_test,
    test_size=0.5
)

The model :

model = tf.keras.Sequential([
    tf.keras.layers.Dense(X.shape[1], activation='relu', input_dim=X.shape[1]),
    tf.keras.layers.Dense(X.shape[1] * .66 + X.shape[1] + 1, activation='relu'),
    tf.keras.layers.Dropout(.10),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

model.compile(
    loss='binary_crossentropy',
    optimizer='adam',
    metrics=['accuracy']
)

lr_sched = step_decay_schedule(initial_lr=1e-4, decay_factor=0.75, step_size=2)
callbacks = [
    lr_sched,
    PrintLR(),
    monitor
]

hist = model.fit(
    X,
    Y,
    epochs=50,
    batch_size=128,
    shuffle=True,
    validation_data=(X_validation, Y_validation),
    callbacks=callbacks
)

And here are some confusion matrix results

50.0 %
['6634', '77', '104', '1114']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['83.67%', '0.97%', '1.31%', '14.05%']

80.0 %
['6691', '20', '283', '935']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.39%', '0.25%', '3.57%', '11.79%']

85.0 %
['6693', '18', '328', '890']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.41%', '0.23%', '4.14%', '11.22%']

90.0 %
['6699', '12', '388', '830']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.49%', '0.15%', '4.89%', '10.47%']

95.0 %
['6705', '6', '507', '711']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.56%', '0.08%', '6.39%', '8.97%']

97.0 %
['6709', '2', '585', '633']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.61%', '0.03%', '7.38%', '7.98%']

99.0 %
['6711', '0', '757', '461']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.64%', '0.00%', '9.55%', '5.81%']

100 %
['6711', '0', '1218', '0']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.64%', '0.00%', '15.36%', '0.00%']

The function for the confusion matrix:

def my_rounder(num, threshold):
    if num >= threshold:
        return 1
    else:
        return 0

def print_conf_matrix_assessment(data, y_expected, threshold, prediction_model):
    y_actual = [[int(my_rounder(x, threshold)) for x in y] for y in prediction_model.predict(data)]
    conf_matrix = confusion_matrix(y_expected, y_actual)
    label_names = ['True Neg', 'False Pos', 'False Neg', 'True Pos']
    result_percentages = ["{0:.2%}".format(value) for value in conf_matrix.flatten() / np.sum(conf_matrix)]
    print()
    print(threshold * 100, "%")
    sample_counts = ["{0:0.0f}".format(value) for value in conf_matrix.flatten()]
    print(sample_counts)
    print(label_names)
    print(result_percentages)

So in general things look pretty good. But as we know if it looks too good there is something wrong and I am trying to figure that out since when I get some production data and put it through the model I get results like that instead:

50.0 %
['2585', '1681', '23', '767']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['51.13%', '33.25%', '0.45%', '15.17%']

60.0 %
['2683', '1583', '30', '760']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['53.07%', '31.31%', '0.59%', '15.03%']

70.0 %
['2782', '1484', '38', '752']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['55.02%', '29.35%', '0.75%', '14.87%']

80.0 %
['2918', '1348', '51', '739']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['57.71%', '26.66%', '1.01%', '14.62%']

85.0 %
['3007', '1259', '59', '731']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['59.47%', '24.90%', '1.17%', '14.46%']

90.0 %
['3129', '1137', '81', '709']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['61.89%', '22.49%', '1.60%', '14.02%']

95.0 %
['3305', '961', '113', '677']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['65.37%', '19.01%', '2.23%', '13.39%']

97.0 %
['3446', '820', '149', '641']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['68.16%', '16.22%', '2.95%', '12.68%']

99.0 %
['3704', '562', '254', '536']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['73.26%', '11.12%', '5.02%', '10.60%']

100 %
['4266', '0', '790', '0']
['True Neg', 'False Pos', 'False Neg', 'True Pos']
['84.38%', '0.00%', '15.62%', '0.00%']

And here we are talking of data that is 2 hours apart. So concept shift is doubtful. The data is on the same scale - using the min max values from the training dataset and using the mean if some value is out of bounds.

Does anyone have any idea what could be the reason for this to happen?