# Questions tagged [linear-algebra]

For questions about the use/aspects/implementation/intuition/mathematical proofs of various Linear Algebra methods used in Machine Learning and AI algorithms.

44 questions
Filter by
Sorted by
Tagged with
21 views

### Analyzing a deep learning model by constructing a Matrix from input and output data

I am currently completing my bachelor’s thesis, and recently my supervisor suggested that I could strengthen my arguments by explaining why deep learning models perform so well in my case. To give you ...
• 101
40 views

• 101
1 vote
178 views

### Why my best fit line is not having a single straight line | Multiple Linear Regression

I am working on Multiple Linear Regression (Multiple variables). I am been able to predict and get a good r2 score. But I am not sure that I understood the part of plotting the best fit line, I can't ...
1 vote
13 views

### Comparing Auto-regressive Encoder-Decoders and Topological Neural Networks

I am interested in what insights can be gained about the mathematical class of auto-regressive encoder-decoders (LLMs), by comparing them to topological neural networks. Specifically, I am looking for ...
497 views

### How to work with multiple embeddings?

This is a conceptual gap that I have concerning embeddings, and would really appreciate some help closing it. I understand when you embed a corpus for, let's say, a question-and-answer task you can ...
78 views

### Is orthogonal initialization still useful when hidden layer sizes vary?

Pytorch's orthogonal initialization cites "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks ", Saxe, A. et al. (2013), which gives as reason for the ...
• 121
47 views

536 views

### Do solving system of linear equations required anywhere in contemporarty deep learning?

Consider the following from Numerical Computation chapter of Deep Learning book Machine learning algorithms usually require a high amount of numerical computation. This typically refers to algorithms ...
• 3,890
2k views

### Is there any difference between affine transformation and linear transformation?

Consider the following statements from A Simple Custom Module of PyTorch's documentation To get started, let’s look at a simpler, custom version of PyTorch’s Linear module. This module applies an ...
• 3,890
1k views

### How can we find the value function by solving a system of linear equations?

I am following the book "Reinforcement Learning: An Introduction" by Richard Sutton and Andrew Barto, and they give an example of a problem for which the value function can be computed ...
• 143
546 views

### What's the difference between a 1d tensor and a 2d tensor with 1 dimension?

I'm doing a TensorFlow tutorial, where they convert an array of the numbers [1,2,3] to a tensor like this: ...
• 167
248 views

### How can the gradient of the weight be calculated in the viewpoint of matrix calculus?

Let $\sigma(x)$ be sigmoid function. Consider the case where $\text{out}=\sigma(\vec{x} \times W + \vec{b})$, and we want to compute $\frac{\partial{\text{out}}}{\partial{w} }.$ Set the dimension as ...
• 99
2k views

### How to express a fully connected neural network succintly using linear algebra?

I'm currently reading the paper Federated Learning with Matched Averaging (2020), where the authors claim: A basic fully connected (FC) NN can be formulated as: $\hat{y} = \sigma(xW_1)W_2$ [...] ...
375 views

### Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

If the derivative is supposed to give the rate of change of a function at that point, then why is the derivative of the softmax layer (a vector) the Jacobian matrix, which has a different shape than ...
32 views

### How to interpret the variance calculation in a Guassian process

I answered another question here about the mean prediction of a GP, but I have a hard time coming up with an intuitive explanation of the variance prediction of a GP. Thew specific equation that I am ...
• 139
154 views

### What exactly is the eigenspace of a graph (in spectral clustering)?

When we find the eigenvectors of a graph (say in the context of spectral clustering), what exactly is the vector space involved here? Of what vector space (or eigenspace) are we finding the ...
1 vote
179 views

1 vote
41 views

### Human intuition behind SVD in case of recommendation system

This does not answer my question. I struggled very hard to understand the SVD from a linear-algebra point of view. But in some cases I failed to connect the dots. So, I started to see all the ...
• 111
285 views

### What do we mean by 'principal angle between subspaces'?

I came across the term 'principal angle between subspaces' as a tool for comparing objects in images. All material that I found on the internet seems to deal with this idea in a highly mathematical ...
1 vote
59 views

### Simplifying Log Loss

I am reading through a paper (https://www.mitpressjournals.org/doi/pdf/10.1162/0891201053630273) where they describe logloss as a ranking function and can be simplified to the margin of the training ...
• 247
6k views

### How do you find the homography matrix given 4 points in both images?

I want to understand the process of finding a homography matrix given 4 points in both images. I am able to do that in python OpenCV, but I wonder how it works behind the scenes. Suppose I have ...
• 185
1 vote
70 views

### Is there any way to apply linear transformations on a vector other than matrix multiplication?

I am trying to optimize the cost function calculation in regression analysis using a non-matrix multiplication based approach. More specifically, I have a point $x = (1, 1, 2, 3)$, to which I want to ...
182 views

### Why MLP cannot approximate a closed shape function?

[TL;DR] I generated two classes Red and Blue on a 2D space. Red are points on Unit Circle and Blue are points on a Circle Ring with radius limits (3,4). I tried to train a Multi Layer Perceptron ...
1 vote
35 views

### CSP Formulation of an algebraic problem

Is anyone able to explain how to do this? I'm not looking for the complete answer, I would settle for a "how to for dummies" explanation of how this is supposed to be solved. I understand ...
• 11
221 views

### Which linear algebra book should I read to understand vectorized operations?

I am reading Goodfellow's book about neural networks, but I am stuck in the mathematical calculus of the back-propagation algorithm. I understood the principle, and some Youtube videos explaining this ...
• 151
1 vote
36 views

### Estimating Baselines using ALS

I am trying to figure out how ALS works when minimizing the following formula: $\\ \\$ \$\text{min}_{\lbrace b_u,b_i \rbrace} \sum_{(u,i)\in \mathcal{K}} (r_{ui} - \bar{r} - b_u - b_i )^2 + \lambda_{...
• 155
269 views

### What does it mean to do multi-dimensional processing with tensors in tensor cores?

In some tweets about NeurIPS 2018, this video from NVIDIA appeared. At around 0.37, she says: If you think about the current computations in our deep learning systems, they are all based on Linear ...
• 161
1 vote
351 views

### Using reinforcement learning to find a preconditioner for linear systems of the form Ax = b

Sparse linear systems are normally solved by using solvers like MINRES, Conjugate gradient, GMRES. Efficient preconditioning, i.e., finding a matrix P such that PAx = Pb is easier to solve than the ...
• 31
897 views

### Does k consistency always imply (k - 1) consistency?

From Russell-Norvig: A CSP is strongly k-consistent if it is k-consistent and is also (k − 1)-consistent, (k − 2)-consistent, . . . all the way down to 1-consistent. How can a CSP be k-consistent ...
• 131