What is the meaning of "energy loss" in PCA?

Question

Recently in a slide in about PCA (Principal Component Analysis) I saw a question: "How much is the data energy loss in PCA?" Then in the next slide I saw somethings that do not understand relation of last steps to the PCA and energy. Although I know that PCA tries to find principle components which have variances in decreasing order and also guess the energy means the total variance after keeping some first principle components and applying it on normalized data; but in this way I think what is described in Wikipedia as cumulative energy content for each eigenvector and what is described here, are more rational and understandable.

Here you can see part of slide that tries to show an example:

Steps are:

1. X as input
2. normalize X (as Z)
3. compute covariance matrix of Z as R
4. eigenvalues of R in descending order as $\lambda$ and their corresponding vectors as V.
5. $S = V\lambda^{\frac{1}{2}}$ where $\lambda^{\frac{1}{2}}$ is element-wise operation.
6. discard third column of S and sore it as S
7. calculate diagonal elements of $S S^{T}$ as var columnar matrix
8. 1 - var(2,1) is the amount of energy loss of PCA. Here it is 1 - 0.8420.

This is an Octave code that you can follow to get the same results:

X=[
7 4 3;4 1 8;6 3 5;8 6 1;8 5 7;
7 2 9;5 3 3;9 5 8;7 4 5;8 2 2];

rows=size(X)(1);
XMean = repmat(mean(X),rows,1);
XStd = repmat(std(X),rows,1);
X_normalized = (X - XMean)./(XStd);

Cov_mat=cov(X_normalized);

[V,lambda] = eigs(Cov_mat) %descending order
S = V * lambda^(0.5);
S1 = S(:,1:2);
var = diag(S1*S1')

Here are what I do not understand:

• What is the definition of energy loss of PCA? If it is relative to what is described in Wikipedia, then how is it relative to the steps that here are described in above slide?
• Where can I found a reference or some examples about these steps?
• Why did we keep only first two columns of S?
• Why 1 - var(2,1) is the amount of energy loss of PCA? Does 2 in 1 - var(2,1) is relative to the first two columns?
• Is 1 - var(2,1) is the amount of energy loss of PCA by using two eigenvectors relative to the two eigenvectors with largest magnitude?
• If we want to calculate energy loss for only using eigenvector that is relative to the largest eigenvalue, do we have to calculate $[S(1,1) S(1,2) S(1,3)]^T [S(1,1) S(1,2) S(1,3)]$ as var and 1 - var(1,1) is the amount of energy loss?

Answer 1

Energy loss in principle component analysis (PCA) is the decrease in the sum of the squares of the coefficients of the scores obtained (so the coordinates for that row of a data set) from the covariance matrix relative to the sum of the squares of the coefficients of original input vectors (the true coordinates if you prefer to express it that way).