Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep?
- Number of heads
- Number of encoder & decoder layers
- Size of transformer model (
- Size of hidden layers
Are there general guidelines like number of decoder layers should be equal to encoder layers? Thank you