1
$\begingroup$

According to the authors of this paper, to improve the performance, they decided to

drop backward pass and using a first-order approximation

I found a blog which discussed how to derive the math but got stuck along the way (please refer to the embedded image below):

  1. Why disappeared in the next line.
  2. How come (which is an Identity matrix)

FOMAML

Update: I also found another math solution for this. To me it looks less intuitive but there's no confusion with the disappearance of 𝜃 as in the first solution. first order MAML

$\endgroup$
  • $\begingroup$ Regarding your 1st question, what is $\theta$? In all other cases, $\theta$ has a subscript. $\endgroup$ – nbro Mar 9 at 15:22
  • $\begingroup$ θ was in the Algorithm 1, but then in the blog it uses both θ and $\theta_{0}$. Maybe they're both "initial model parameter" and the notation was not consistent. I'm working on understanding it too .. $\endgroup$ – Long Mar 10 at 9:40
2
$\begingroup$

$\nabla_{\theta_{i-1}} \theta_{i-1} = \mathbf{I}$ in a similar way that $\frac{d f}{dx} = 1$ for $f(x) = x$. Strictly speaking, $\mathbf{I}$ should be a vector of $1s$ with the same dimensionality as $\theta_{i-1}$, but they are probably abusing notation here and putting such a vector as the diagonal elements of a matrix. Alternatively (actually, the most likely reason!), they are computing the partial derivative of $\theta_{i-1}^j$ with respect to $\theta_{i-1}^k$, for all $k$, for all $j$, which will make up an identity matrix.

Regarding your first question, $\nabla_{\theta} \theta_{0}$ probably becomes 1, but I am not familiar enough with the math of this paper to tell you why. Maybe it's because $\nabla_{\theta} \theta_{0}$ actually means $\nabla_{\theta_0} \theta_{0}$. I would need to dive into it.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ we don't know what to differentiate it with I guess. I think it should not be included in the chain rule. It is probably a random initialization. $\endgroup$ – DuttaA Mar 9 at 15:57
  • $\begingroup$ @DuttaA Well, I guess that what you say is consistent with my explanation. If you look at that part of the formula, you see terms where the gradient with respect to the previous parameters is taken of the current parameters. However, initially, we only have $\theta_0$ (i.e. no previous parameters), so the derivative of $\theta_0$ with respect to itself is $1$. $\endgroup$ – nbro Mar 9 at 16:00
  • $\begingroup$ I updated my question with a second solution. Please have a look and give some comment about it. It seems like in this case we don't need to worry about the $\theta_{0}$ anymore. $\endgroup$ – Long Mar 10 at 9:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.