While I was studying the equations for the computation inside GRU and LSTM units, I realized that although the different gates have different Weight matrices, their overall structure is the same. They are all dot products of a weight matrix and their inputs, plus bias, followed by a learned gating activation. Now, the difference between computation depends on the weight matrices being different from each other, that is, those weight matrices are specifically for specializing in the particular tasks like forgetting/keeping etc.
But these matrices are all initialized randomly, and it seems that there's no special tricks in the training scheme to make sure these weight matrices are learned in a manner that the associated gates specialize in their desired tasks. They are all random matrices that kept getting updated with gradient descent.
So how does, for example, a forget gate learn to function as a forgetting unit? Same question applies to others as well. Am I missing a part of the training for these networks? Can we ever say that these units learn truly disentangled functions from each other?