I'm trying to understand these two points from an article:

  • Models with large variables i.e weight matrices. As a consequence such models have correspondingly large gradients and optimizer states. The activations (intermediate outputs from the model layers) tend to be relatively small (depends on the batch size). Typically fully connected networks and RNNs fall under this category.
  • Models with small weights but large activations. CNNs and transformers tend to fall under this category.

This is my attempt to understand it:

My current understanding is that large variables would mean large matrices, and the corresponding gradient matrices are of the same size. In fully connected neural network the activation of one layer would be the same size as the next layer. So activation should usually be 1 dimension.

Now the confusing part: A model with small weights can have large activations -- I think this is because the same (small) weight matrices(e.g. 3x3 in CNN) are reused. While the activations are of dimension $(n-(k-1)) \times (n-(k-1))$, where $k$ is the width of the kernel.

Is my understanding correct?

  • $\begingroup$ Could you please provide the link to the article to have more context? $\endgroup$
    – nbro
    Commented May 16, 2022 at 8:26
  • $\begingroup$ Sure. Thanks for your help in advance. This is the GitHub link, the two quoted points can be found at the start of the ipynb file. I was trying to understand when could this package be applied. @nbro $\endgroup$
    – NRain
    Commented May 16, 2022 at 9:25


You must log in to answer this question.