Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep?

  • Number of heads
  • Number of encoder & decoder layers
  • Size of transformer model (d_model in Pytorch)
  • Size of hidden layers

Are there general guidelines like number of decoder layers should be equal to encoder layers? Thank you


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.