# What's the definition of “singularity” in the context of neural networks?

The paper Skip connections eliminate singularities explains the use of skip connections to break the singularity in deep networks, but I have not fully understood what a singularity is.

Any easy-understanding explanation?

Firstly, I invite you to take this answer with a grain of salt (and possibly suggest edits) as I am not specifically familiar with the issue of singularities myself. Also, I appologise for all the links being at the bottom of the answer, I am unable to reuse links throughout my answer due to my reputation on this particular exchange.

This appears to refer to singular points in Algebraic Geometry. At singular points, no singlar tagent can be defined using the usual methods. This is as opposed to regular (i.e non-singular points), where one (and only one) tangent can be regularly defined.

In simpler terms, and particularly with respect to neural networks, it refers to the points on a hypersurface (e.g. a line, plane, surface, etc) at which the gradient/slope is ambigious.

I've taken example of this from the wikipedia page on singular points.

In the above image, the hyperspace is a line. On this line, there is a singularity at (0,0), where the line crosses itself. At this point, there are two slopes (one going up, and the other going down). There is no way to choose between the slopes without applying arbatrary rules, so the gradient is ambigous.

These singularities are problematic for some of the algorithms used in deep learning, such as Gradient Descent, which uses the the derivative of objective functions (e.g. loss functions) to search for the weights/parameters that yield the lowest error.

## Gradient Descent In This Example

For anyone unfamiliar with gradient descent, the problem with singularaties can be understood with the following example:

Say that the above line represents a path. Starting at the top right, you want to find the lowest point on the path. At any time, all you can know is the slope at the current point. You can turn left or right freely, but will never move upwards, so you will stop as soon as you reach the bottom of a valley.

Initially moving down and to the left, you will eventually reach the singularity and need to choose to go either left or right. Because you have no information about the path beyond your current location, you have no way of knowing which way leads to the bottom of the path.

If you choose at random, you have a 50% chance of getting stuck at in the valley at the bottom of the loop on the left of the path.

You could impose a rule like "always follow the gradient most different to the current gradient", then you'd go right, and find the true lowest point. However, if the path was slightly different (e.g. if it curved up at (0.25, 0.25) this would be bad, as the correct path would then be on the left.

In practice, there is no way of knowing which way is better without a deep understanding of the hypersurface of your objective functions (and if you knew that much about the objective function, you likely wouldn't need to use machine learning in the first place).

## Research/Resources

I discovered this in order to answer this questions so, as justification, here is the process and resources I used to come to this conclusion:

I dug through the references in the paper you linked. Of particular relevance was Amari et al. (2006), which mentions Hironaka’s theorem of singularity resolution.