0
$\begingroup$

I am trying to make a model that uses a Transformer to see the relationship between several data vectors, but the order of the data is not relevant in this case, so I am not using the Positional Encoding.

Since the performance of models using Transformers is quite improved with the use of this part, do you think that if I remove that part I am breaking the potential of Transformers or is it correct to do so?

$\endgroup$
1
  • $\begingroup$ Maybe you could make a little example of the inputs and how they should be given to the transformer, to make it clear why the order of the data should not be relevant. $\endgroup$
    – dexteritas
    Nov 15 '21 at 15:55
1
$\begingroup$

Positional Encodings in Transformers exist to give the model some information about the position of the embedding. This makes sense in fields like NLP or Time Series Data, since the position(order) matters in this case.

However, since you say that order of the data is not relevant in your use case, positional encoding would not be necessary.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.