I had a specific problem where I had a leave-one-subject-out cross-validation scheme that was a little complex in terms of scoring.
Specifically, I had 21 subjects, and for each subject I had between ca. 5 and 10 data points, and if any one of them were correctly classified as class 1, that would correspond to 100% recall/sensitivity for that subject; whereas all points that were incorrectly classified as class 1 despite being class 0 were counted as false positives. My metrics were therefore sensitivity and false positive rate, with sensitivity being more important. Therefore, this was a specific scoring and cross-validation situation that would make it harder for feature selection methods like RFE or Lasso to work properly. In fact, those methods were not effective.
My solution was to implement a fairly exhaustive (and rudimentary) selection method, namely by:
Removing combinations of 1, 2, or 3 features at each iteration Choosing the best result based on sensitivity, and then false positive rate if the sensitivity was the same for multiple combinations At the next iteration, with the reduced set, either remove combinations of 1,2 or 3 features again, or add back features that had been removed in previous steps (while not adding back exactly the previously removed features, and also not removing for example combinations of 2 features if only 1 had been removed, because those combinations had been tested in the previous step), based on the highest sensitivity/fpr again. Continuing with iterations until it's not improving anymore This is a bit of a heuristic, since you're not going to get the perfect combination of features, and potentially a little prone to overfitting as you're testing so many things that you might get caught in the small local maxima of accuracy.
In any case, I'm publishing a paper detailing the algorithm in which this all fits, and I'm wondering whether or not to include this method. Is this something that already exists and/or already has a name? Is it very stupid and will make me look stupid? Or is it actually good and worth including in the paper?
Thank you!
