Return to Question

I was using PCA in my whole dataset (and after split to training, validation and test), but after some researchs I found out that is wrong way to do. Then I have few questions:

-Are there some articles/references explain why is the wrong way?

-How can I transform the validation/test set?

Steps to do PCA (from https://www.sciencedirect.com/science/article/pii/S0022460X0093390X):

1. zero mean

$$\mu = \frac{1}{M}\sum_{i=1}^{M} x_{i}$$

where x is my training set

2. centering (variance)

$$S^{2} = \frac{1}{M}\sum_{i=1}^{M} (x_{i}-\mu)^{T}(x_{i}-\mu)$$

3. use (1) and (2) to transform my original training dataset

$$x_{new} = \frac{1}{\sqrt{M}} \frac{(x_{i} - \mu)}{S}$$

4. calculate covariance matrix (actually correlation matrix)

$$C= x_{new}^T x_{new}$$

5. take the k-eigenvectors (/phi) from covariance matrix and defined the new space for my new dimension training set (where k are the principal components that I choose acording my variance)

$$ x_{new dim} = x_{new}\phi$$

Ok, then I have my new dimensional training dataset after PCA (till here its right according to other papers that I have read). The question is: What I have to do now for my validation/testing set?

just the equation below?

$$y_{new dim} = y\phi $$

where y is my (for exemple) validation original dataset?

Can someone explain the right thing to do?

Thanks!!! :)

I was using PCA on my whole dataset (and, after that, I would split it into training, validation, and test datasets). However, after a little bit of research, I found out that this is the wrong way to do it. 

I have few questions:

1. Are there some articles/references that explain why is the wrong way?

2. How can I transform the validation/test set?

Steps to do PCA (from https://www.sciencedirect.com/science/article/pii/S0022460X0093390X):

1. zero mean

$$\mu = \frac{1}{M}\sum_{i=1}^{M} x_{i}$$

where x is my training set

2. centering (variance)

$$S^{2} = \frac{1}{M}\sum_{i=1}^{M} (x_{i}-\mu)^{T}(x_{i}-\mu)$$

3. use (1) and (2) to transform my original training dataset

$$x_{new} = \frac{1}{\sqrt{M}} \frac{(x_{i} - \mu)}{S}$$

4. calculate covariance matrix (actually correlation matrix)

$$C= x_{new}^T x_{new}$$

5. take the k-eigenvectors (/phi) from the covariance matrix and defined the new space for my new dimension training set (where k are the principal components that I choose according to my variance)

$$ x_{new dim} = x_{new}\phi$$

Ok, then I have my new dimensional training dataset after PCA (till here it's right, according to other papers that I have read). 

The question is: *What I have to do now for my validation/testing set? Just the equation below?

$$y_{new dim} = y\phi $$

where y is my (for example) validation original dataset?

Can someone explain the right thing to do?

I was using PCA in my whole dataset (and after split to training, validation and test), but after some researchs I found out that is wrong way to do. Then I have few questions:

-Are there some articles/references explain why is the wrong way?

-How can I transform the validation/test set?

Steps to do PCA (from https://www.sciencedirect.com/science/article/pii/S0022460X0093390X):

1. zero mean

$$\mu = \frac{1}{M}\sum_{i=1}^{M} x_{i}$$

where x is my training set

2. centering (variance)

$$S^{2} = \frac{1}{M}\sum_{i=1}^{M} (x_{i}-\mu)^{T}(x_{i}-\mu)$$

3. use (1) and (2) to transform my original training dataset

$$x_{new} = \frac{1}{\sqrt{M}} \frac{(x_{i} - \mu)}{S}$$

4. calculate covariance matrix

$$C= x_{new}^T x_{new}$$

5. take the k-eigenvectors (/phi) from covariance matrix and defined the new space for my new dimension training set (where k are the principal components that I choose acording my variance)

$$ x_{new dim} = x_{new}\phi$$

Ok, then I have my new dimensional training dataset after PCA (till here its right according to other papers that I have read). The question is: What I have to do now for my validation/testing set?

just the equation below?

$$y_{new dim} = y\phi $$

where y is my (for exemple) validation original dataset?

Can someone explain the right thing to do?

Thanks!!! :)

I was using PCA in my whole dataset (and after split to training, validation and test), but after some researchs I found out that is wrong way to do. Then I have few questions:

-Are there some articles/references explain why is the wrong way?

-How can I transform the validation/test set?

Steps to do PCA (from https://www.sciencedirect.com/science/article/pii/S0022460X0093390X):

1. zero mean

$$\mu = \frac{1}{M}\sum_{i=1}^{M} x_{i}$$

where x is my training set

2. centering (variance)

$$S^{2} = \frac{1}{M}\sum_{i=1}^{M} (x_{i}-\mu)^{T}(x_{i}-\mu)$$

3. use (1) and (2) to transform my original training dataset

$$x_{new} = \frac{1}{\sqrt{M}} \frac{(x_{i} - \mu)}{S}$$

4. calculate covariance matrix (actually correlation matrix)

$$C= x_{new}^T x_{new}$$

5. take the k-eigenvectors (/phi) from covariance matrix and defined the new space for my new dimension training set (where k are the principal components that I choose acording my variance)

$$ x_{new dim} = x_{new}\phi$$

Ok, then I have my new dimensional training dataset after PCA (till here its right according to other papers that I have read). The question is: What I have to do now for my validation/testing set?

just the equation below?

$$y_{new dim} = y\phi $$

where y is my (for exemple) validation original dataset?

Can someone explain the right thing to do?

Thanks!!! :)