the best choice to reduce a features vector

Question

i have 1200 features highly correlated , and i want to reduce those number of features so the best choice is use feature selection or dimensionality reduction? and which method is the best in this case. feature selection can work with features highly correlated ??

Answer 1

• Feature selection -- the case in which the features are highly correlated is the prototypical case in which you want to select a subset of independent features that allows for an equal performance. However keep in mind that, exactly because they are correlated, there are in general multiple subsets of features that can achieve the same goal (i.e., if x_1 is highly correlated with x_2, you can drop either of them).

• Dimensionality reduction -- if your features are only linearly correlated, then a linear dimensionality reduction technique (e.g., PCA) will tackle your problem. If however the dependence is more than linear (and unknown), dimensionality reduction will construct new features that are a complex combination of the original ones, hence you will get to the same problem, just in a reduced-dimensional space.

TLDR; As usual in ML, it depends on what you need to do. If it is merely a matter of reducing dimensions in order to speed up calculations, either method is valid. If you need instead to select features for further [causal?] analysis, you need to lean for the first approach. But, again, be aware of multiple equivalent sets (e.g., see this paper: https://arxiv.org/abs/1611.03227)