# What is $USV^T$ in the context of word embeddings?

Here is an excerpt from the notes of the first lecture of the course CS224n: Natural Language Processing with Deep Learning.

3 SVD Based Methods

For this class of methods to find word embeddings (otherwise known as word vectors), we first loop over a massive data set and accumulate word co-occurrence counts in some form of a matrix $$X$$ and then perform Singular Value Decomposition on $$X$$ to get a $$USV^T$$ decomposition. We then use the rows of $$U$$ as the word embeddings for all words in our dictionary. Let us discuss a few choices of $$X$$.

What does $$USV^T$$ refer to in this context?

## 1 Answer

$$USV^T$$ refers to the result of the singular value decomposition (SVD).

An $$m \times n$$ matrix $$X$$ can be written with the help of three matrices

$$X = USV^T,$$

where $$U$$ is an $$m \times m$$ unitary matrix, $$S$$ is a diagonal $$m \times n$$ matrix with real entries called singular values, and $$V$$ is a unitary $$n \times n$$ matrix. The $$^T$$ is the Hermitian transpose. SVD has applications, e.g, in optimization problems, principal component analysis, etc. Wikipedia has quite a long article.