0
$\begingroup$

In case I had a prediction model and decided to add a PCA step prior to the model, is it theoretically possible/impossible that the number of output dimensions that is better for all tests may perform worse than the model without PCA?

My question comes from the fact that I want to add a PCA step prior to a model and hyperparameterize the PCA output dimension from 1 to N (N being the number of dimensions in the original dataset) and I wanted to know if there is any theoretical basis that there is no case in which performing this previous step could have a worse performance than the previous model.

Especially my doubt comes if the best PCA case from a selection of dimensions from 1-N is always better than the best case without PCA.

$\endgroup$
  • $\begingroup$ What do you mean by "number of output dimensions that is better". How is an "output dimension" better than another? Please, edit your post to clarify this! $\endgroup$ – nbro Apr 30 at 15:44
1
$\begingroup$

PCA works well where data sample space is linear. If data sample space is not linear or it is manifold data then model without PCA may perform better than model using PCA.

Manifold Data

In the given image you can see, data is manifold. In this type of data, PCA, which is based on projection technique does not work well. That's why we use manifold learning technique to handle these cases.

| improve this answer | |
$\endgroup$
1
$\begingroup$

PCA can make models worse, imagine data points scattered along two elongated parallel rectangles. The axis with the greatest variation will be parallel to the rectangles but doesn't provide any benefit in classifying the points.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.