What is $USV^T$ in the context of word embeddings?

Question

Here is an excerpt from the notes of the first lecture of the course CS224n: Natural Language Processing with Deep Learning.

3 SVD Based Methods

For this class of methods to find word embeddings (otherwise known as word vectors), we first loop over a massive data set and accumulate word co-occurrence counts in some form of a matrix $X$ and then perform Singular Value Decomposition on $X$ to get a $USV^T$ decomposition. We then use the rows of $U$ as the word embeddings for all words in our dictionary. Let us discuss a few choices of $X$.

What does $USV^T$ refer to in this context?

Answer 1

$USV^T$ refers to the result of the singular value decomposition (SVD).

An $m \times n$ matrix $X$ can be written with the help of three matrices

$$X = USV^T,$$

where $U$ is an $m \times m$ unitary matrix, $S$ is a diagonal $m \times n$ matrix with real entries called singular values, and $V$ is a unitary $n \times n$ matrix. The $^T$ is the Hermitian transpose. SVD has applications, e.g, in optimization problems, principal component analysis, etc. Wikipedia has quite a long article.