2
$\begingroup$

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep?

  • Number of heads
  • Number of encoder & decoder layers
  • Size of transformer model (d_model in Pytorch)
  • Size of hidden layers

Are there general guidelines like number of decoder layers should be equal to encoder layers? Thank you

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.