What is the formal definition for manifold in artificial intelligence?

Question

We come across the word "manifold" in artificial intelligence, especially in the domains where learning is done based on data instances.

What is the formal definition for manifold?

Answer 1

Manifold is basically a geometric object where every small region can be mapped to a euclidean space(means manifold is locally euclidean). Think of a donut, here any small region can be mapped to a euclidean space shown in this image:

In this above picture, $M$ is the manifold, $\phi_\alpha,\phi_\alpha$ is the mapping function, $U_\alpha, U_\beta$ is two open sets(small local regions). This donut is an example of a manifold. Similarly, we can think of circles, spheres, paraboloids, $\mathbb{R}^2$, $\mathbb{R}^3$, etc. as a manifold because they all satisfy the above criteria(they are locally euclidean).

Now, the question is why we are interested in manifolds in machine learning. In many machine learning applications, the data we interpret is laying on a manifold or non-Euclidean domain. For example, in astrophysics the observational data often time lies on a spherical domain. If we want we perform convolution over this spherical manifold to extract features, we can't just apply 2D convolution since we have to take account of parallel transport, gauges, symmetries, etc. Similarly, we may want to perform convolution over more complex shapes like those figures to extract features.

There are methods like guage equivalent mesh CNN, geodesic CNN, etc to deal with such kind of data distribution.

Graphs also lie on a non-Euclidean domain since the distance between any two nodes is not a straight line we have to travel through the graph and count the number of edges to measure distance. There are many applications where data lies on a graph, for example, drug-drug interaction, community detection, molecule structure, friendship network, recommendation system, traffic forecasting, etc.

To perform convolution over graphs we have methods like ChebNet, GraphSAGE, graph attention network, etc.

Notes:

1) Parallel transport: One basic problem occurs when we try to compare two vectors of two different points over the same manifold is that those two vectors belong to different Euclidean spaces (see the first figure), thus we can not directly compare them. Parallel transport provides a mechanism to move vectors over a manifold and analysis them. But note that parallel transport depends on the path means the result of the parallel transport is path-dependent.

2) Guage: Guage is like a measurement apparatus to specify the tangent vector on the tangent space of a manifold.

References:

1. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics

2. To learn about geometric deep learning this is very good website, they also contain extraordinary video lectures on GDL

Note that: I intentionally skipped the rigorous mathematical definition of a manifold while trying to convey the underlying meaning. Please, let me know if you want to know more about open sets, closed sets, topological spaces, topological manifold, charts, atlas, etc.

Answer 2

The definition is the same as in Mathematics and, I suppose, elsewhere:

it is a topological space such that the vicinity of each point is homeomorphic to a disk in $\mathbb{R}^n$ (note, that dimension has to be the same for all points $x$). This requirement is important, since not every imaginable geometric object satisfies this requirement:

• Sphere is a manifold, since one can draw a tangent plane in the vicinity of any point. It is "locally flat" and even we humans see Earth flat, since its radius of curvature is much greater than the visible distance (let the proponents of the flat Earth theory forgive me).

• Two intersecting lines are not a manifold, since for any point, except for the intersection we have a 1-dimensional space and for the intersection point, there are two non-collinear vectors belonging to them.

Natural examples emerging in Machine Learning are images, videos, or arbitrary data. One usually treats, say, an image, as an object in the $\mathbb{R}^{H \times W \times 3}$, where $H$ is the height, $W$ - width of the image, and $3$ - number of colors. But in fact, only a small subset of all objects in this high-dimensional space are real images, and they belong to some manifold of a lower dimension.

It is a non-trivial question to tell what exactly the true dimensionality of data is. For MNIST, it is claimed that it is $3$ (instead of $28 \times 28 = 784$).

As a good material on this topic, I recommend this lecture from the recent workshop.

Scikit-learn has a nice exposition as well.