I read the AlBERT and saw that its architecture used "Parameter Sharing" among layers of the encoder. They mentioned that this was done to save model space, make fewer training parameters and also because it was observed that most BERT layers learned almost the same thing. My question is that if we consider that the first layer in AlBERT approximates a function (say F(X)), then can it be said that overall AlBERT just applies that function repeatedly on the input to generate the output?

ŷ = F<sub>1</sub>(F<sub>2</sub>(F<sub>2</sub>(...F<sub>*n*</sub>(x))

  • where x is the tokenized input, n is the number of layers in the encoders of AlBERT and ŷ is the output from the final layer?

Here I assume that both the multi-head attention and feed-forward network together approximate F(x).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.