"A significant computational expense in MAML comes.from the use of second derivatives when backpropagating the meta-gradient through the gradient operator in the meta-objective (see Equation (1)). On MiniImagenet, we show a comparison to a first-order approximation of MAML, where these second derivatives are omitted. Note that the resulting method still computes the meta-gradient at the post-update parameter values $θ^ 0_i$, which provides for effective meta-learning. Surprisingly however, the performance of this method is nearly the same as that obtained with full second derivatives, suggesting that most of the improvement in MAML comes from the gradients of the objective at the post-update parameter values, rather than the second order updates from differentiating through the gradient update.".
$$\theta \leftarrow \theta \; - \beta \nabla_\theta \sum_{\mathcal{T}_i\thicksim p (\mathcal{T})} \mathcal{L_{T_i}}(\mathcal{f}_{\theta^{{'}}_i}) \tag{1} $$
"A significant computational expense in MAML comes.from the use of second derivatives when backpropagating the meta-gradient through the gradient operator in the meta-objective (see Equation (1)). On MiniImagenet, we show a comparison to a first-order approximation of MAML, where these second derivatives are omitted. Note that the resulting method still computes the meta-gradient at the post-update parameter values $θ^ 0_i$, which provides for effective meta-learning. Surprisingly however, the performance of this method is nearly the same as that obtained with full second derivatives, suggesting that most of the improvement in MAML comes from the gradients of the objective at the post-update parameter values, rather than the second order updates from differentiating through the gradient update.".
"A significant computational expense in MAML comes.from the use of second derivatives when backpropagating the meta-gradient through the gradient operator in the meta-objective (see Equation (1)). On MiniImagenet, we show a comparison to a first-order approximation of MAML, where these second derivatives are omitted. Note that the resulting method still computes the meta-gradient at the post-update parameter values $θ^ 0_i$, which provides for effective meta-learning. Surprisingly however, the performance of this method is nearly the same as that obtained with full second derivatives, suggesting that most of the improvement in MAML comes from the gradients of the objective at the post-update parameter values, rather than the second order updates from differentiating through the gradient update.".
$$\theta \leftarrow \theta \; - \beta \nabla_\theta \sum_{\mathcal{T}_i\thicksim p (\mathcal{T})} \mathcal{L_{T_i}}(\mathcal{f}_{\theta^{{'}}_i}) \tag{1} $$
You have several questions, so I'll refer you to the source; which answers all questions with one short answer.
According to: "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks" it is initialized in Class MAML:
"How does MAML inner loop optimization work?"
It is initialized by xavier_initializer_conv2d from TensorFlow, as explained in "Understanding the difficulty of training deep feedforward neural networks", Xavier Glorot, and Yoshua Bengio, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR 9:249-256, 2010.
A method to alleviate the risk of overfitting for gradient-based meta-learning is explained in: "Regularizing Meta-Learning via Gradient Dropout" - April 13 2020, by Tseng, Chen, Tsai, Liu, Lin, and Yang:
"[they] aim to find model parameters that are sensitive to changes in the task, such that small changes in the parameters will produce large improvements on the loss function of any task drawn from $p(T)$, when altered in the direction of the gradient of that loss.".
Combined, those methods shorten the time taken to complete each pass.
In the line:
FLAGS.datasource == 'omniglot' or FLAGS.datasource == 'miniimagenet':
self.loss_func = xent
See pages 6 and 7:
"A significant computational expense in MAML comes.from the use of second derivatives when backpropagating the meta-gradient through the gradient operator in the meta-objective (see Equation (1)). On MiniImagenet, we show a comparison to a first-order approximation of MAML, where these second derivatives are omitted. Note that the resulting method still computes the meta-gradient at the post-update parameter values $θ^ 0_i$, which provides for effective meta-learning. Surprisingly however, the performance of this method is nearly the same as that obtained with full second derivatives, suggesting that most of the improvement in MAML comes from the gradients of the objective at the post-update parameter values, rather than the second order updates from differentiating through the gradient update.".
You have several questions, so I'll refer you to the source; which answers all questions with one short answer.
According to: "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks" it is initialized in Class MAML:
It is initialized by xavier_initializer_conv2d from TensorFlow, as explained in "Understanding the difficulty of training deep feedforward neural networks", Xavier Glorot, Yoshua Bengio Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR 9:249-256, 2010.
According to: "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks" it is initialized in Class MAML:
"How does MAML inner loop optimization work?"
It is initialized by xavier_initializer_conv2d from TensorFlow, as explained in "Understanding the difficulty of training deep feedforward neural networks", Xavier Glorot and Yoshua Bengio, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR 9:249-256, 2010.
A method to alleviate the risk of overfitting for gradient-based meta-learning is explained in: "Regularizing Meta-Learning via Gradient Dropout" - April 13 2020, by Tseng, Chen, Tsai, Liu, Lin, and Yang:
"[they] aim to find model parameters that are sensitive to changes in the task, such that small changes in the parameters will produce large improvements on the loss function of any task drawn from $p(T)$, when altered in the direction of the gradient of that loss.".
Combined, those methods shorten the time taken to complete each pass.
In the line:
FLAGS.datasource == 'omniglot' or FLAGS.datasource == 'miniimagenet':
self.loss_func = xent
See pages 6 and 7:
"A significant computational expense in MAML comes.from the use of second derivatives when backpropagating the meta-gradient through the gradient operator in the meta-objective (see Equation (1)). On MiniImagenet, we show a comparison to a first-order approximation of MAML, where these second derivatives are omitted. Note that the resulting method still computes the meta-gradient at the post-update parameter values $θ^ 0_i$, which provides for effective meta-learning. Surprisingly however, the performance of this method is nearly the same as that obtained with full second derivatives, suggesting that most of the improvement in MAML comes from the gradients of the objective at the post-update parameter values, rather than the second order updates from differentiating through the gradient update.".