When we are training a neural network, we are going to determine the embedding size to convert the categorical (in NLP, for instance) or continuous (in computer vision or voice) information to hidden vectors (or embeddings), but I wonder if there are some rules to set the size of it?
In most cases, seems that embedding dim is chosen empirically, by trial and error.
Older papers in NLP used 300 conventionally https://petuum.medium.com/embeddings-a-matrix-of-meaning-4de877c9aa27. More recent papers used 512, 768, 1024.
One of the factors, influencing the choice of embedding is the way you would like different vectors to correlate with each other. In high dimensional space with probability 1, chosen at random vectors would be approximately mutually orthogonal. Whereas in the low dimensions and case of many different classes, many vectors will have dot product, significantly different from 0.
I think, that if one expects, that many vectors have to be correlated then the dimension shouldn't be very high. And otherwise, if each of the possible keys in the embedding is expected to produce a different, unrelated vector, than dimensionality is expected to be large.
I get an answer from this book: Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and MLOps.
If we’re in a hurry, one rule of thumb is to use the fourth root of the total number of unique categorical elements while another is that the embedding dimension should be approximately 1.6 times the square root of the number of unique elements in the category, and no less than 600.