I'm working on building a Hopfield Transformer using the github code from the paper (https://github.com/ml-jku/hopfield-layers/tree/master/hflayers) to forecast a timeseries dataset with 48 variables, 20 past timesteps, and 4 future timesteps. I'm then working to extract the attention weight matrix.
I'm noticing that the attention weight matrix has rows that are all the same.
The way I'm building this Transformer is with a source dataset of shape [176, 48, 20] and a target dataset of shape [176, 48, 4] where 176 is the batch size, 48 is the "sequence" length, and 20 (or 4) is the embedding dimension. In the encoder, I first embed and perform sequential encodings on the 20 timesteps to obtain a [176, 48, 1280] tensor, which gets sent to the Hopfield Encoder from the github (or the hopfield attention layer, either way it works the same), and then to the Decoder, with a linear layer that reshapes the source and target data shapes to match.
**My model does appear to significantly outperform a vanilla transformer. However when extracting the attention weight matrix, I'm noticing that the rows appear the same: **
Is this supposed to happen, or am I missing something about Hopfield Transformer functionality in the context of my data shape? I'm not sure what to do to resolve this - I'm happy to provide code if it helps!
