Using tensor networks as machine learning models

Question

Tensor networks (check this paper for a review) are a numerical method originally introduced in condensed matter physics to model complex quantum systems. Roughly speaking, such systems are described by a very high-dimensional tensor (where the indices take a number of values scaling exponentially with the number of system constituents) and tensor networks provide an efficient representation of the latter as an outer product and contraction of many low-dimensional tensors.

More recently, a specific kind of tensor network (called Matrix Product State in physics) found interesting applications in machine-learning through the so-called Tensor-Train decomposition (I do not know of a precise canonical reference in this context, so I will abstain from citing anything).

Now, over the last few years, several works from the physics community seemed to push for a generalized use of tensor networks in machine learning (see this paper, a second one and a third one and this article from Google AI for context). As a physicist, I am glad to learn that tools initially devised for physics may find interdisciplinary applications. However, at the same time, my critical mind tells me that from the machine learning research community's perspective, these results may not look that intriguing. After all, machine learning is now a very established field and it takes probably more than a suggestion for a new machine learning model and a basic benchmarking on a trivial dataset (as the MNIST one) -which is what the papers essentially do in my humble opinion- to attract any attention in the area. Besides, as I believe to know, there already exists quite a solid body of knowledge on tensor analysis techniques for machine learning (e.g. tensor decompositions), which may cast doubt on the originality of the contribution.

I would therefore be very curious to have the opinion of machine learning experts on this line of research: is it really an interesting direction to look into, or is it just about surfing on the current machine learning hype with a not-so-serious proposal?

Answer 1

I do not feel comfortable proclaiming that I would be a machine learning expert. But I want to point out that there is indeed interest in applying tensor networks in machine learning settings. Let me highlight three particular settings in the following.

They can be used to find the governing equations behind dynamical systems analogous to the SINDy algorithm. The reasoning is that the dynamics of your system may not be sparse but may indeed have a low-rank structure. (See this paper for a comparison of sparse and low-rank structures and this paper for a discussion about why low-rank matrices so often appear in big data.) The resulting modification of SINDy is the appropriately named MANDy algorithm. But if your dynamical law exhibits additional structure (despite being low-rank) then this structure is not used. (Requiring you to collect unnecessary samples.) This work tackles this issue by allowing you to specify the structure that you expect your law to exhibit.

Another area of interest is the application of tensor networks as a means of compression. Tensor networks can be used to approximate multivariate functions. Either directly (see this article or this article for further details) or by representing their high-dimensional coefficient tensor (see e.g. here). The last paper also highlights where such functions may occur: when solving stochastic or parametric PDEs. "What does this have to do with machine learning?" you may rightfully ask. These PDEs are hard to solve and it is often easier to minimize a residual than to apply a Galerkin method. The results can be shown to be equivalent with high probability. (See this paper for a proof of this statement and an application to some stochastic PDEs or this paper for an application to an optimal control problem.)

Moreover, there is the problem of tensor completion - a generalization of matrix completion to higher dimensions. (This and this are references for algorithms.) These methods find application in data science where they can be used to decompose or denoise data. (Unfortunately, I don't have a reference with examples but I remember a presentation where these methods where used on EEG signals.)

Additionally, I recently found this interesting symposium covering even more potential applications of tensor networks in machine learning: https://itsatcuny.org/calendar/quantum-inspired-machine-learning