Is accuracy of a machine learning algorithm independent of the features?

Question

I’ve been experimenting on several datasets and found something very strange while implementing ML.
I’ll Explain after the code…

import numpy as np
from sklearn import datasets
iris = datasets.load_iris()

# 4 features in np array - 150 rows

case = 1        # change cases to see variation

if case == 1:   # first feature deleted
    iris.data = np.delete(iris.data,0, 1)

if case == 2:   # first 2 features deleted
    iris.data = np.delete(iris.data,0, 1)
    iris.data = np.delete(iris.data,0, 1)

if case == 3:   # first 3 features deleted (1 feature left)
    iris.data = np.delete(iris.data,0, 1)
    iris.data = np.delete(iris.data,0, 1)
    iris.data = np.delete(iris.data,0, 1)

if case == 4:   # only second feature deleted from np array
    iris.data = np.delete(iris.data,1, 1)

if case == 5:   # only third feature deleted from np array
    iris.data = np.delete(iris.data,2, 1)

if case == 6:   # only last feature deleted from np array
    iris.data = np.delete(iris.data,3, 1)

# print iris.data
# exit()

from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB()
pred = gnb.fit(iris.data, iris.target).predict(iris.data)
# pred = gnb.fit(iris.data, iris.target).predict(test_data)

from sklearn.metrics import accuracy_score
print accuracy_score(iris.target, pred)

I’m using the basic fisher iris dataset from sklearn, it has 150 rows and 4 columns (features).
Using training data as test data.

So i tried to remove a few features and see if accuracy changed. And I thought it would.
But till case 1, 2, and 3, I removed 1, 2 and 3 features respectively and there was no change in the accuracy. It stayed 96%.

Then on running cases 4,5 and 6, The accuracy changed. Why?

On comparing case 2 and 4,
Both have second feature removed from the dataset, so clearly, removing second feature is responsible for change in accuracy (as seen in case 4)
So why does it not change in case 2?
Just because it had first feature removed too? to balance out the second? (If that were true, case 1 would have given different accuracy)
Why does accuracy not change in first 3 cases, but it does in the last 3 cases?

Is ML dependent on the order in which features are feeded to the algorithm?

What am I missing here?

Would be great if someone could clear this doubt.

Thanks!

Answer 1

This is a result of overfitting, I think. Your model is allowed to explain the phenomenon using any combination of variables, whether or not such a combination constitutes a true signal.

I am not familiar with the details of a NBC, but I suspect that the model has some bias/offset/constant variable, that can be combined with your input features.

Thusly, what you see is just an expression that the classifier has been able to combine each set of features and the constant variables in such a way that they perfectly describe the phenomenon. These combinations will be non-linear, so the logic as to how dependent the model is on any parameter is not a simple "it follows"-reasoning.

However, if you evaluate OUT of sample, you will see a much better reflection of how useful individual features are, and how bad the performance becomes when you remove the first three features.

For example, do a K-folds test (say, K=10) of the six different models, preserving 20% of the dataset as test-data.

Here you would likely see the following rankings:

(4, 5, 6, 1 ~ equal) > 2 > 3