I found myself scratching my head when I read the following phrase in the paper Visualizing the Loss Landscape of Neural Nets:

To remove this scaling effect, we plot loss functions using filter-wise normalized directions. To obtain such directions for a network with parameters $\theta$, we begin by producing a random Gaussian direction vector $d$ with dimensions compatible with $\theta$. Then, we normalize each filter in $d$ to have the same norm of the corresponding filter in $\theta$. In other words, we make the replacement $d_{i,j} \leftarrow d_{i,j} \| d_{i,j}\| \| \theta_{i,j}\| $

I'm completely unclear what the authors are referring to when they refer to the filters of the vector $d$ in weight space. As far as I can tell, the vector $d$ is a standard vector in weight space ($W$) with a number of components equal to the number of changeable weights in the network. In my opinion, it could be said that each layer in the network can be visualized as a vector in weight space ($\theta_{i}$) with:

$$\theta = \sum_{i}\theta_{i}$$

and then maybe these vectors $\theta_{i}$ are called filters? But how this would have anything to do with the random vector $d$, generated in this space, remains a complete mystery to me.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.