What is geometric deep learning?

Question

What is geometric deep learning (GDL)?

Here are a few sub-questions

• How is it different from deep learning?
• Why do we need GDL?
• What are some applications of GDL?

Answer 1

To complete the first answer that is rather graph oriented, I will write a little about deep learning on manifolds, which is quite general in terms of GDL thanks to the nature of manifolds.

Note that the description of GDL through the explanation of what are DL on graphs and manifolds, in opposition to DL on euclidean domains, comes from the 2017 paper Geometric deep learning: going beyond Euclidean data (this paper is excellent at clarifying both the intuition and the mathematics of what I'm writing).

1. In case you don't know what a manifold is

As the previously cited paper puts it:

Roughly, a manifold is a space that is locally Euclidean. One of the simplest examples is a spherical surface modeling our planet: around a point, it seems to be planar, which has led generations of people to believe in the flatness of the Earth. Formally speaking, a (differentiable) d-dimensional manifold X is a topological space where each point x has a neighborhood that is topologically equivalent (homeomorphic) to a d-dimensional Euclidean space, called the tangent space.

Good other not-so-technical explanation on stats.stackexchange

Other Wikipedia examples to develop not too abstract understanding

Very shortly put, it's an interesting mathematical set on which to work (different kinds exist, see papers at the end of this answer for DL related manifolds uses). By work, you can typically understand that you constrain the neural net parameters to the manifold you chose (e.g. training with parameters constrained on a hypersphere, among the geomstats paper examples).

Your data can also be represented thanks to a practical manifold. For example, you can choose to work on images and videos by representing the samples using Symmetric Positive Definite (SPD) matrices (see this paper), the space of SPD matrices being a manifold itself.

2. Why bother learning on manifolds ?

Defining a clearer/better adapted set (understand that it's a sort of constraint!) on which to learn parameters and features can make it simpler to formally understand what your model is doing, and can lead to better results. I see it as a part of the effort of deep learning formalization. One could say you're looking for the best information geometry for your task, the one that best captures the desirable data distribution properties.To develop this intuition consider the solar system analogy for manifold learning of this Kaggle kernel:

Perhaps a good analogy here is that of a solar system: the surface of our planets are the manifolds we're interested in, one for each digit. Now say you're on the surface of the earth which is a 2-manifold and you start moving in a random direction (let's assume gravity doesn't exist and you can go through solid objects). If you don't understand the structure of earth you'll quickly find yourself in space or inside the earth. But if you instead move within the local earth (say spherical) coordinates you will stay on the surface and get to see all the cool stuff.

This analogy reminds us of the spherical surface planet model from Bronstein's paper already quoted above. This paper also describes a typical case for which manifolds are interesting: where graphs (the other example of GDL/DL on non euclidean data) are better at handling data from social or sensor networks, manifolds are good at modeling 3D objects endowed with properties like color texture in computer vision.

3. Regarding deep neural networks on manifolds

I would advise reading the geomstats associated paper, which does a great job at showing what it is and how it can be used, along with example codes (e.g. MNIST on hyperspheres manifold example code here). This library implements manifolds and associated metrics on Keras. The choice of metrics is essential to understand the point of working on manifolds: it's because you need to work on an adapted mathematical set (ie with the right properties) with an adapted distance definition (so that the measure actually means something when considering the problem you're trying to solve) that you switch to working on manifolds.

If you want to dive in the details and examples of deep learning on manifolds here are some papers:

• A Riemannian Network for SPD Matrix Learning: new backpropagation to learn SPD matrices on Riemannian manifolds
• Learning a Robust Representation via a Deep Network on Symmetric Positive Definite Manifolds: using SPD matrices to aggregate convolutional features

4. Why Riemannian manifolds ?

TL;DR: you need a metric to do machine learning (otherwise, how could you evaluate how much you actually learned !)

Still based on Bronstein's paper:

On each tangent space, we define an inner product [...]. This inner product is called a Riemannian metric in differential geometry and allows performing local measurements of angles, distances, and volumes. A manifold equipped with a metric is called a Riemannian manifold.

5. What's the relation between a Riemannian manifold and a Euclidean space ?

Still based on Bronstein's paper:

a Riemannian manifold can be realized as a subset of a Euclidean space (in which case it is said to be embedded in that space) by using the structure of the Euclidean space to induce a Riemannian metric.

I leave the details to the paper, otherwise this answer will never end.

6. Answers to questions in comments

Will only answer once I think I've found a relatively well-argued answer, so won't answer everything at once.

• Isn't manifold learning just a way of dimensionality reduction?

I don't think so, it isn't just that. I haven't seen any dimensional reduction constraint (yet ?) in the papers I've read (cf. geomstats again).

In the hypersphere/MNIST geomstats code example, you can see the chosen manifold dimension hypersphere_dimension = 17. Since we're working with MNIST data I guess this would mean a dimension reduction in this particular case. I admit I would need to check exactly what that dimension implies on the neural net architecture, I haven't discussed my understanding of this yet.

Disclaimer

I'm still developing a more rigorous mathematical understanding of manifolds, and shall update this post to make additional necessary clarifications: exactly what can be considered as a manifold in a traditional deep learning context, why do we use the word manifold when speaking about the hidden state of auto-encoders (see the previously cited Kaggle kernel that quotes Goodfellow's book on this). All of this if the perfectly clear answer doesn't show up here before !

Answer 2

The article Geometric deep learning: going beyond Euclidean data (by Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, Pierre Vandergheynst) provides an overview of this relatively new sub-field of deep learning. It answers all the questions asked above (and more). If you are familiar with deep learning, graphs, linear algebra and calculus, you should be able to follow this article.

What is geometric deep learning (GDL)?

This article describes GDL as follows

Geometric deep learning is an umbrella term for emerging techniques attempting to generalize (structured) deep neural models to non-Euclidean domains such as graphs and manifolds.

So, the inputs to these GDL models are graphs (or representations of graphs), or, in general, any non-Euclidean data. To be more concrete, the input to these models (e.g. graph neural networks) are e.g. feature vectors associated with the nodes of the graphs and matrices which describe the graph structure (e.g. the adjacency matrix of the graphs).

Why are e.g. graphs non-Euclidean data?

A graph is a non-Euclidean structure because e.g. distances between nodes are not well defined. Yes, you can have graphs with weights associated with the edges, but not all graphs have this property.

What classes of problems does GDL address?

In GDL, there are two classes of problems that are often tackled:

1. characterise the structure of the data (e.g. of a graph)
2. analyse functions defined on a given non-Euclidean domain

These classes of problems are related, given that the structure of the graph imposes certain properties on the functions that can be defined on it. Furthermore, these properties of these functions can also convey information about the structure of the graph.

What are applications of GDL?

An example of an application where this type of data (graphs) arises is in the context of social networks, where each user can be associated with a vertex of the social graph and the characteristics (or features) of each user (e.g. number of friends) can be represented as a feature vector (which can then be associated with the corresponding vertex of a graph). In this context, the goal might e.g. be to determine different groups of users in the social network (i.e. clustering).

Why can't we simply use deep learning methods (like CNNs) when the data is non-Euclidean?

There are several problems that arise when dealing with non-Euclidean data. For example, operations like convolution are not (usually) defined on non-Euclidean data. More concretely, the relative position of nodes is not defined on graphs (but this would be required to perform the usual convolution operation): in other words, it is meaningless to talk about a vertex that is e.g. on the left of another vertex. In practice, it means that we can't simply use the usual CNN when we are given non-Euclidean data. There have been attempts to generalise the convolution operation to graphs (or to approximate it). The field is still quite new, so there will certainly be new developments and breakthroughs.

Answer 3

I didn't read the paper in depth, but one example of where assumptions of Euclidean space are made in the design of the networks are with ConvNets in image processing.

Specifically, Euclidean spaces are transformationally invariant, meaning that $d(a,b) = d(a+c,b+c)$. Each convolution layer iterates over the image with a certain amount of stride, which guarantees a certain amount of translational invariance in exchange for a smaller size of network parameters.

ConvNets would not naively work for a dataset which comes from, say, a graph, because its not clear how to stride over it, as a possible example.