I found myself scratching my head when I read the following phrase in the paper Visualizing the Loss Landscape of Neural Nets:
To remove this scaling effect, we plot loss functions using filter-wise normalized directions. To obtain such directions for a network with parameters $\theta$, we begin by producing a random Gaussian direction vector $d$ with dimensions compatible with $\theta$. Then, we normalize each filter in $d$ to have the same norm of the corresponding filter in $\theta$. In other words, we make the replacement $d_{i,j} \leftarrow d_{i,j} \| d_{i,j}\| \| \theta_{i,j}\| $
I'm completely unclear what the authors are referring to when they refer to the filters of the vector $d$ in weight space. As far as I can tell, the vector $d$ is a standard vector in weight space ($W$) with a number of components equal to the number of changeable weights in the network. In my opinion, it could be said that each layer in the network can be visualized as a vector in weight space ($\theta_{i}$) with:
$$\theta = \sum_{i}\theta_{i}$$
and then maybe these vectors $\theta_{i}$ are called filters? But how this would have anything to do with the random vector $d$, generated in this space, remains a complete mystery to me.
