1
$\begingroup$

I was using PCA in my whole dataset (and after split to training, validation and test), but after some researchs I found out that is wrong way to do. Then I have few questions:

-Are there some articles/references explain why is the wrong way?

-How can I transform the validation/test set?

Steps to do PCA (from https://www.sciencedirect.com/science/article/pii/S0022460X0093390X):

  1. zero mean

$$\mu = \frac{1}{M}\sum_{i=1}^{M} x_{i}$$

where x is my training set

  1. centering (variance)

$$S^{2} = \frac{1}{M}\sum_{i=1}^{M} (x_{i}-\mu)^{T}(x_{i}-\mu)$$

  1. use (1) and (2) to transform my original training dataset

$$x_{new} = \frac{1}{\sqrt{M}} \frac{(x_{i} - \mu)}{S}$$

  1. calculate covariance matrix (actually correlation matrix)

$$C= x_{new}^T x_{new}$$

  1. take the k-eigenvectors (/phi) from covariance matrix and defined the new space for my new dimension training set (where k are the principal components that I choose acording my variance)

$$ x_{new dim} = x_{new}\phi$$

Ok, then I have my new dimensional training dataset after PCA (till here its right according to other papers that I have read). The question is: What I have to do now for my validation/testing set?

just the equation below?

$$y_{new dim} = y\phi $$

where y is my (for exemple) validation original dataset?

Can someone explain the right thing to do?

Thanks!!! :)

$\endgroup$
2
$\begingroup$

for the first point I'm very sorry that I cannot give you any literature on this, but I might be able to explain you, why you don't take PCA on both datasets independently.

Principal components analysis is simply a transformation of your data into another (less dimensional) coordinate system. The axis for your new coordinate system are defined by the principal components (i.e. eigen-vectors) of your covariance matrix.

Since you will train your machine learning algorithm in the domain that is generated by the PCA, your test data must be exactly in the same domain. So as you said, you use exactly the same transformation for the test data as for the training data, i.e. $y_{newdim} = y \phi $. Of course if you applied standardization to your training data, you have to apply the same standardization to your test data. So you need to store the mean $\mu_x$ and the standard deviation $S_x$ and also standardize your test data y to \begin{equation} y_{standardized} = \dfrac{y_i - \mu_x}{S_x} \end{equation}

Note here that you have an error in your standardization formula (you do not need to divide by $M$).

The point is that the principal components of your test data $\phi_{y}$ would not match the principal components of your training data $\phi_{x}$. Thus the transformations from original space into PCA-space $\Phi_x(u)$ and $\Phi_y(u)$ would diverge and similar data points in the original space might be far away in the PCA-representation and vice versa. This is why you generate the mapping $x_{newdim} = \Phi_x(x) = x\phi$ and apply it also on the test data.

I hope I could make it clear to you.

Best

$\endgroup$
  • $\begingroup$ Thanks @adriculteur. Than I do not need substract the zero-mean and std deviation that I did for the training set from the validation/test set? just multiply the validation/test dataset with the eigenvectors and it is done? $\endgroup$ – LVoltz Oct 30 '18 at 13:51
  • $\begingroup$ Sorry if I had not been explicit on that. The test data has to undergo exactly the same transformation as the training data. I will edit my answer accordingly. $\endgroup$ – adriculteur Oct 30 '18 at 14:54
  • $\begingroup$ sorry I wrote wrong, actually is srqt(M), as the article that I wrote as references, but it does not matter because it will cancel in the thrd equation. Anyway thank you so much for you aswer, it helps me a lot!!! Best regards! $\endgroup$ – LVoltz Oct 30 '18 at 15:40
  • $\begingroup$ I am happy my answer helped you, but actually the standard score is calculated as you did in equation (3) but without the 1/sqrt(M). Just look at the equation in my answer or here: en.wikipedia.org/wiki/Standard_score#Calculation_from_raw_score $\endgroup$ – adriculteur Oct 30 '18 at 15:47
  • $\begingroup$ Hi again! You are right, as I was following the article and the both equations have M or /sqrt(M) and then they call the covariance matrix as the correlation matrix when the difference between covariance matrix and correlation matrix is because the correlation matrix is a normalized form from covariance matrix. :) $\endgroup$ – LVoltz Oct 30 '18 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.