I am currently completing my bachelor’s thesis, and recently my supervisor suggested that I could strengthen my arguments by explaining why deep learning models perform so well in my case.

To give you some context, I am essentially attempting to learn, in a supervised manner, an output vector given an input vector.

Specifically, what I aim to achieve with deep learning is to learn the inverse operator of a matrix transformation, as illustrated by the following equation:

$$ \vec{x} = A^{-1}\cdot \vec{y} $$

So, what the deep learning is trying to do is the following: $$ \vec{x} = DL(\vec{y}, \theta), $$ where $\theta$ are the weights of the model.

I thought of in trying to conduct a spectral analysis of the deep learning model to compare its spectral analysis (distribution of eigenvectors) to the spectral analysis of the inverse of matrix 𝐴 A. However, since the deep learning model is non-linear, I cannot perform a traditional linear spectral analysis. Given this limitation, I have come up with an alternative idea:

Let's assume that there exists a matrix $C$, such that it can map the input vectors and the output vectors yielded by the deep learning model. My idea is now to conduct the spectral analysis on this constructed matrix (e.g. by the method of the least-squares or SVD) and then state that the spectral analysis of this matrix $C$ is approximate to the spectran analysis of the matrix $A{^-1}$.

I have tried this empirical validation and it seems to confirm my assumption. Somehow, the deep learning model, even though its non linear, it has transformed the input y into a input x, so that there exists an approximate linearity between these vectors and which can be captured with the matrix $C$.

Since the deep learning model performs very well and the spectral analysis between matrix $C$ and $A^{-1}$ are similar and the opposite with a bad deep learning model, could i affirm that the deep learning model seems to perform well, because the linearity that its produce seems to be in the spectral realm of the matrix $A^{-1}$ ?

The thing about this is that its a big assumption and i haven't found literature on this...

I would be very glad to here some insights from you

Best regards

  • $\begingroup$ Your neural network is overfitting to the distribution of matrix you are feed it: matrix inversion is $O(n^3)$, your neural network is a finite set of $O(n^2)$ operations, so you definitely cannot teach a plain neural network to invert "any" matrix $\endgroup$
    – Alberto
    Commented Jun 11 at 10:22


You must log in to answer this question.

Browse other questions tagged .