Sensible integer embedding/encoding for distinguishing elements of a set?

Question

I am trying to train a model that takes in a set of feature vectors (which comes with an ID to uniquely identify elements of the set) and outputs a target for each element in the set (in a permutation-equivariant manner). It seems like a self-attention based model would work well here (because there's reason to believe that the target depends on the features of other elements).

However, I need help coming up with a sensible encoding for the integer ID that comes along with the feature vector. For context, I have at 3561 unique integers IDs (0 being the "out of vocabulary" ID (such as for elements not seen during training) and 1 to 3560 for the elements seen during training). Intuitively, it seems like a bad idea to just provide the integer itself concatenated to the feature vector.

The goal is to concatenate the integer embedding with the feature vector and train self-attention on this set.

The constraints for the encoding/embedding of integer IDs are:

1. No elements are similar: There are no physical/temporal relationships here, just a bag of elements. So, something like positional encoding as used in the Transformer paper might not be a good choice because there's no reason to believe that element #45 and element #46 have any special relationship compared to element #46 and #100.

2. They should be deterministic and more importantly, not trained. Trainable embeddings would only result in the embeddings changing during training which is undesirable.

3. Low dimensional: Definitely not more than, say, 50 dimensions (so one-hot encodings are not acceptable here).

---

My best idea right now is to use spherical codes. For example, unit vectors in say 24 dimensions that are distributed as far from each other as possible. Since the kissing number in 24 dimensions is 196,560 - 3560 unit-vectors can be easily placed in a way such that the pair-wise cosine similarity for any two vectors is quite low. The out-of-vocabulary embedding vector would be the zero-vector (at the origin).

Would this be sensible? I have heard that positional encoding had some desirable properties with respect to linear transformations. Would the kind of integer embedding I am imagining based on equidistant unit vectors cause problems as the self-attention module is largely a linear operation.

I am probably overthinking this and there likely exist sensible ways of embedding/encoding integer IDs that I simply haven't heard of. For all I know, just providing the binary representation would work just as well as any other embedding.

Answer 1

As you have asserted that the id is not meaningful of itself, it probably doesn't matter how you encode it.

I would recommend the following, in order:

• Don't encode the id. It is not clear that you need it when you already have the feature vector. At the very least you should train the model initially without the id to get a baseline and find out out if using the id has any benefit.

• Use a secure hash function, such as SHA1, which will "randomly" assign each id a byte array (e.g. effectively 16 bytes or 128 bits of pseudo-randomness using the data as an initial value or seed). You can then take the first N bits of each hash as a short vector of 1s and 0s. If N is small you may get collisions amongst the ids, but this may not be a practical issue, if only one or two ids are special enough to affect the target beyond the feature vector. About 30 bits should be enough to encode your ids with a low chance of collision. The advantage of this approach is you can get SHA1 as off-the-shelf function and the rest is easy.

• The spherical mapping function or similar that you can think of. Don't set yourself up with lots of work though, unless you expect to be able to re-use the encoding in a more meaningful way.