# How does PCA work when we reduce the original space to 2 or higher-dimensional space?

How does PCA work when we reduce the original space to a 2 or higher-dimensional space? I understand the case when we reduce the dimensionality to $$1$$, but not this case.

$$\begin{array}{ll} \text{maximize} & \mathrm{Tr}\left( \mathbf{w}^T\mathbf{X}\mathbf{X}^T\mathbf{w} \right)\\ \text{subject to} & \mathbf{w}^T\mathbf{w} = 1\end{array}$$

You might want to have a look at the wikipedia article of PCA, where it says:

"The $$k$$th component can be found by subtracting the first $$k − 1$$ principal components from $$\mathbf{X}$$:"

$$\hat{\mathbf{X}}_k = \mathbf{X} - \sum_{s=1}^{k-1}\mathbf{X}\mathbf{w}_s\mathbf{w}_s^T$$

Then you repeat the process to find the next component:

$$\mathbf{w}_k = \arg\max \mathbf{w}^T\mathbf{\hat{X}}^T_k\mathbf{\hat{X}}_k\mathbf{w}$$ $$\text{s.t. } \mathbf{w}_k^T\mathbf{w}_k = 1$$

You can also understand the logic from the view of constrained optimisation. Introduce a Lagrange function: $$\mathcal{L} = \text{Tr} (w^{T} X X^{T} w) - \lambda w^{T} w$$ And take the derivative with respect to $$w$$: $$\frac{\partial \mathcal{L}}{\partial w} = 2 (X X^{T} - \lambda) w$$ For the general case of dimension $$\geqslant 1$$ $$w$$ is a set of vectors $$w = (w_1 w_2 \ldots w_n)$$. This expression vanishes, if for some index $$i$$ $$w_i$$ is an of eigenvector of $$XX^{T}$$ with the eigenvalue $$\lambda_i$$, and all other components are set to zero. In other words, stationary points are the eigenvectors of $$X X^{T}$$.

The condititon $$w^T w = 1$$ imposes the orthogonality condition on the eigenvectors. In fact, going back to the initial functional, one sees, that $$w_i X X^{T} w_j = \lambda_j w_i^{T} w_j = 0$$ for $$i \neq j$$. Therefore, we have finally: $$\mathcal{L} =\sum \lambda_i - \lambda$$ Which is maximized for any $$k \geq 1$$, by taking $$k$$ largest eigenvalues.

• 1.) Is the duality gap zero for such functions? 2.) The $L$ is minimised for k largest Eigen values. And the major concern I haven't seen this type of formulation without building the dual problem, can you link a resource? (Not doubting, but I am interested)
– user9947
Sep 1, 2020 at 23:16