What does it mean by "gradient flow" in the context of neural networks?

Question

Several research papers and textbooks (e.g. this) contain the phrase "gradient flow" in the context of neural networks.

I am confused about whether it has any rigorous and formal way of understanding or not. What is the flow referring to here?

Answer 1

It has. Gradient flow or more generally flow is a well known concept in maths. Say we have a function $f:\mathbb R^n \longrightarrow \mathbb R^n$ and a function $\theta:[0,\infty)\longrightarrow \mathbb R^n$ such that the ODE $$ \partial_t \theta(t) = f(\theta(t)) $$ exists for any initial choice $\theta(0)\in\mathbb R^n$ uniquely. Then i think the naming comes from the following informal description. Lets imagine the case $n=2$ then $\mathbb R^2$ is a plane. Now you drop $\theta(0)$ somewhere on this plane and wait some time $t$ to see where the $\theta(0)$ will flow to. You can also draw the trajectories for multiple starting points $\theta(0)$ which will often give you an image that looks like a flow of a liquid.

In the case of the ANN $f$ would be the negative gradient of the costfunction with respect to the paramter-vector $\theta$ and gradient descent would be an approximation of the gradient flow.

Answer 2

Here is my idea of what that means: Gradient flow is an abstract term to describe properties of the gradient. The gradient is calculated by propagating the error backwards through the networks, therefore it kind of flows from the last to the first layer. Depending on network architecture and loss function the flow can behave differently.

One popular kind of undesirable gradient flow is the vanishing gradient. It refers to the gradient norm being very small, i.e. the parameter updates are very small which slows down/prevents proper training. It often occurs when training very deep neural networks. Residual connections can help, because they bypass operations that reduce gradient magnitude.

The exact opposite would be exploding gradients where your gradient-norm is very large which leads to unstable training and the weights inside the network can't follow a stable trajectory to the optimum. This often occurs in the context of recurrent neural networks.

Here is a nice article on vanishing/exploding gradients.

A general way to phrase this is that the gradient reveals properties about the loss function which you are optimizing (as it is the derivative). A noisy or heavily oscillating loss function usually implies an undesirable gradient flow.

Thinking about these properties plays an important role in network and loss function design. For example, the root function $\sqrt{\cdot}$ has a very large derivative when approaching zero, so having a $\sqrt{\cdot}$ in the network or the loss function can lead to exploding gradients.