I was using PCA on my whole dataset (and, after that, I would split it into training, validation, and test datasets). However, after a little bit of research, I found out that this is the wrong way to do it.

I have few questions:

  1. Are there some articles/references that explain why is the wrong way?

  2. How can I transform the validation/test set?

Steps to do PCA (from https://www.sciencedirect.com/science/article/pii/S0022460X0093390X):

  1. zero mean

$$\mu = \frac{1}{M}\sum_{i=1}^{M} x_{i}$$

where x is my training set

  1. centering (variance)

$$S^{2} = \frac{1}{M}\sum_{i=1}^{M} (x_{i}-\mu)^{T}(x_{i}-\mu)$$

  1. use (1) and (2) to transform my original training dataset

$$x_{new} = \frac{1}{\sqrt{M}} \frac{(x_{i} - \mu)}{S}$$

  1. calculate covariance matrix (actually correlation matrix)

$$C= x_{new}^T x_{new}$$

  1. take the k-eigenvectors (/phi) from the covariance matrix and defined the new space for my new dimension training set (where k are the principal components that I choose according to my variance)

$$ x_{new dim} = x_{new}\phi$$

Ok, then I have my new dimensional training dataset after PCA (till here it's right, according to other papers that I have read).

The question is: *What I have to do now for my validation/testing set? Just the equation below?

$$y_{new dim} = y\phi $$

where y is my (for example) validation original dataset?

Can someone explain the right thing to do?


1 Answer 1


for the first point I'm very sorry that I cannot give you any literature on this, but I might be able to explain you, why you don't take PCA on both datasets independently.

Principal components analysis is simply a transformation of your data into another (less dimensional) coordinate system. The axis for your new coordinate system are defined by the principal components (i.e. eigen-vectors) of your covariance matrix.

Since you will train your machine learning algorithm in the domain that is generated by the PCA, your test data must be exactly in the same domain. So as you said, you use exactly the same transformation for the test data as for the training data, i.e. $y_{newdim} = y \phi $. Of course if you applied standardization to your training data, you have to apply the same standardization to your test data. So you need to store the mean $\mu_x$ and the standard deviation $S_x$ and also standardize your test data y to \begin{equation} y_{standardized} = \dfrac{y_i - \mu_x}{S_x} \end{equation}

Note here that you have an error in your standardization formula (you do not need to divide by $M$).

The point is that the principal components of your test data $\phi_{y}$ would not match the principal components of your training data $\phi_{x}$. Thus the transformations from original space into PCA-space $\Phi_x(u)$ and $\Phi_y(u)$ would diverge and similar data points in the original space might be far away in the PCA-representation and vice versa. This is why you generate the mapping $x_{newdim} = \Phi_x(x) = x\phi$ and apply it also on the test data.

I hope I could make it clear to you.


  • $\begingroup$ Thanks @adriculteur. Than I do not need substract the zero-mean and std deviation that I did for the training set from the validation/test set? just multiply the validation/test dataset with the eigenvectors and it is done? $\endgroup$
    – LVoltz
    Oct 30, 2018 at 13:51
  • $\begingroup$ Sorry if I had not been explicit on that. The test data has to undergo exactly the same transformation as the training data. I will edit my answer accordingly. $\endgroup$ Oct 30, 2018 at 14:54
  • $\begingroup$ sorry I wrote wrong, actually is srqt(M), as the article that I wrote as references, but it does not matter because it will cancel in the thrd equation. Anyway thank you so much for you aswer, it helps me a lot!!! Best regards! $\endgroup$
    – LVoltz
    Oct 30, 2018 at 15:40
  • $\begingroup$ I am happy my answer helped you, but actually the standard score is calculated as you did in equation (3) but without the 1/sqrt(M). Just look at the equation in my answer or here: en.wikipedia.org/wiki/Standard_score#Calculation_from_raw_score $\endgroup$ Oct 30, 2018 at 15:47
  • $\begingroup$ Hi again! You are right, as I was following the article and the both equations have M or /sqrt(M) and then they call the covariance matrix as the correlation matrix when the difference between covariance matrix and correlation matrix is because the correlation matrix is a normalized form from covariance matrix. :) $\endgroup$
    – LVoltz
    Oct 30, 2018 at 15:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .