Do solving system of linear equations required anywhere in contemporarty deep learning?

Question

Consider the following from Numerical Computation chapter of Deep Learning book

Machine learning algorithms usually require a high amount of numerical computation. This typically refers to algorithms that solve mathematical problems by methods that update estimates of the solution via an iterative process, rather than analytically deriving a formula to provide a symbolic expression for the correct solution. Common operations include optimization (finding the value of an argument that minimizes or maximizes a function) and solving systems of linear equations. Even just evaluating a mathematical function on a digital computer can be difficult when the function involves real numbers, which cannot be represented precisely using a finite amount of memory.

The paragraph clearly mentions that solving system of linear equations is a common operation in machine learning. I just know that solving system of linear equations is useful in reinforcement learning and some basic algorithms of machine learning including regression.

Is solving system of linear equations useful anywhere in deep learning?

I think that we use them nowhere since optimization is the only algorithm generally used in deep learning.

Answer 1

I guess a first distinction should be made between deep learning as a whole or deep learning as architecture. I think the paragraph you quote refers to solving systems of linear equations as a simple operation involved in deep learning generically. And this is definitely the case, when training a deep model we're always solving systems of linear equations, think about the way weights and biases are applied to an input:

$W*x + b$

this is in itself a system of linear equations. Then, moving away from training and model architectures, in deep learning there is still a massive use of dimensionality reduction techniques involved in pre or post processing of the data like SVD or PCA that also consists in solving systems of linear equations (an we could add any matrix factorization technique, relevant especially in the early methods for word embedding training before the advent of transformers).

Answer 2

Well the paragraph you quote says "Machine learning algorithms ...", so it's not restricted to deep learning.

When considering Machine Learning in general there is the case of Gaussian Process Regression where to fit the models a matrix needs to be inverted (which is equivalent to solving a linear system of equations), check equations 2.7 and 2.8 of chapter 2 of the standard GPR book