Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep?

  • Number of heads
  • Number of encoder & decoder layers
  • Size of transformer model (d_model in Pytorch)
  • Size of hidden layers

Are there general guidelines like number of decoder layers should be equal to encoder layers? Thank you



You must log in to answer this question.