Understanding the derivation of the first-order model-agnostic meta-learning

According to the authors of this paper, to improve the performance, they decided to

drop backward pass and using a first-order approximation

I found a blog which discussed how to derive the math but got stuck along the way (please refer to the embedded image below):

1. Why $\nabla_{\theta}\theta_{0}$ disappeared in the next line.
2. How come $\nabla_{\theta_{i-1}}\theta_{i-1}=I$ (which is an Identity matrix)

Update: I also found another math solution for this. To me it looks less intuitive but there's no confusion with the disappearance of 𝜃 as in the first solution.

• Regarding your 1st question, what is $\theta$? In all other cases, $\theta$ has a subscript. – nbro Mar 9 '20 at 15:22
• θ was in the Algorithm 1, but then in the blog it uses both θ and $\theta_{0}$. Maybe they're both "initial model parameter" and the notation was not consistent. I'm working on understanding it too .. – Long Mar 10 '20 at 9:40

$$\nabla_{\theta_{i-1}} \theta_{i-1} = \mathbf{I}$$ in a similar way that $$\frac{d f}{dx} = 1$$ for $$f(x) = x$$. Strictly speaking, $$\mathbf{I}$$ should be a vector of $$1s$$ with the same dimensionality as $$\theta_{i-1}$$, but they are probably abusing notation here and putting such a vector as the diagonal elements of a matrix. Alternatively (actually, the most likely reason!), they are computing the partial derivative of $$\theta_{i-1}^j$$ with respect to $$\theta_{i-1}^k$$, for all $$k$$, for all $$j$$, which will make up an identity matrix.
Regarding your first question, $$\nabla_{\theta} \theta_{0}$$ probably becomes 1, but I am not familiar enough with the math of this paper to tell you why. Maybe it's because $$\nabla_{\theta} \theta_{0}$$ actually means $$\nabla_{\theta_0} \theta_{0}$$. I would need to dive into it.
• @DuttaA Well, I guess that what you say is consistent with my explanation. If you look at that part of the formula, you see terms where the gradient with respect to the previous parameters is taken of the current parameters. However, initially, we only have $\theta_0$ (i.e. no previous parameters), so the derivative of $\theta_0$ with respect to itself is $1$. – nbro Mar 9 '20 at 16:00
• I updated my question with a second solution. Please have a look and give some comment about it. It seems like in this case we don't need to worry about the $\theta_{0}$ anymore. – Long Mar 10 '20 at 9:42