Questions tagged [linear-algebra]
For questions about the use/aspects/implementation/intuition/mathematical proofs of various Linear Algebra methods used in Machine Learning and AI algorithms.
38
questions
0
votes
0
answers
13
views
Linear Discriminant Analysis on a transformed space
Let $S$ be a finite subset of a $\mathbb{R}^k$ partitioned into $N$ subsets $S_1, \ldots, S_N$ and let $n_j = |S_j|$. The between-groups sum of squares of the partition is defined as
$$bSS(S_1,\ldots, ...
- 101
1
vote
2
answers
121
views
Why my best fit line is not having a single straight line | Multiple Linear Regression
I am working on Multiple Linear Regression (Multiple variables). I am been able to predict and get a good r2 score. But I am not sure that I understood the part of plotting the best fit line, I can't ...
1
vote
0
answers
11
views
Comparing Auto-regressive Encoder-Decoders and Topological Neural Networks
I am interested in what insights can be gained about the mathematical class of auto-regressive encoder-decoders (LLMs), by comparing them to topological neural networks.
Specifically, I am looking for ...
- 123
1
vote
0
answers
93
views
How to work with multiple embeddings?
This is a conceptual gap that I have concerning embeddings, and would really appreciate some help closing it.
I understand when you embed a corpus for, let's say, a question-and-answer task you can ...
- 11
2
votes
0
answers
32
views
Is orthogonal initialization still useful when hidden layer sizes vary?
Pytorch's orthogonal initialization cites "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
", Saxe, A. et al. (2013), which gives as reason for the ...
- 121
0
votes
1
answer
42
views
Shape of biases in Transformer's Feedforward Network
In transformer network (Vaswani et al., 2017), the feedforward networks have equation:
$$\mathrm{FNN}(x) = \max(0, xW_1 + b_1) W_2 + b_2$$
where $x \in \mathbb{R}^{n \times d_\mathrm{model}}$, $W_1 \...
- 23
0
votes
1
answer
136
views
How Does The Scaled Dot Product's Dimensions Work Out In Mult-Head Attention?
I don't understand how self-attention works with batched values for the $Q \times K^T $ step. According to the diagram below (assume 1 head), once we get past the first 3 linear steps, we arrive at ...
- 21
0
votes
1
answer
34
views
Is $i$ indexing the first or second dimension in $\mathbf{x}_i$, where $\mathbf{x} \in \mathbb{R}^{n\times d}$?
I was reading the following notes on the math behind transformers and was confused about what $\mathbf{x}_i$ is? If $\mathbf{x} \in \mathbb{R}^{n\times d}$, then is the $i$ indexing the $n$ or the the ...
- 23
1
vote
2
answers
199
views
Encoding data in which the order of features does not matter
My input to the model is a set of features that I encode in the form of five vectors of the same size consisting only of 0 and 1. I now want to combine them into one vector in such a way that their ...
- 173
1
vote
1
answer
140
views
How does using complex weights in a neural network affect performance?
If you switch a neural network from real weights to complex weights, you're roughly doubling the size of the network, and increasing the computational load by a factor of 2 to 4. My question is, in ...
- 150
0
votes
1
answer
66
views
Does $(\langle w, x \rangle + b) = ||x - x'||$ hold?
Currently, I am trying to understand the mathematics of SVM's using the textbook 'Mathematics for Machine Learning' by Deisenrot et. al.
On page 375, they define the distance between a point $x$ and ...
1
vote
0
answers
163
views
Where do the characteristics of self-attention come into play in Linformer's proof that self-attention is low rank?
In Linformer's proof that self-attention is low rank in their paper, I don't see how it doesn't generalize to every matrix. They don't utilize any specifics of self-attention (the entire proof feels ...
- 2,349
1
vote
1
answer
64
views
Why does $I_N + D^{-\frac{1}{2}}AD^{-\frac{1}{2}}$ have eigenvalues in the range [0, 2]?
In Semi-supervised classification with Graph Convolutional Networks, I am unable to understand a few things.
Given an undirected graph having
adjacency matrix $A$,
degree matrix $D_{ii} = \sum_j A_{...
- 35
2
votes
2
answers
379
views
Do solving system of linear equations required anywhere in contemporarty deep learning?
Consider the following from Numerical Computation chapter of Deep Learning book
Machine learning algorithms usually require a high amount of numerical computation. This typically refers to algorithms ...
- 3,581
3
votes
2
answers
1k
views
Is there any difference between affine transformation and linear transformation?
Consider the following statements from A Simple Custom Module of PyTorch's documentation
To get started, let’s look at a simpler, custom version of PyTorch’s
Linear module. This module applies an ...
- 3,581
3
votes
2
answers
911
views
How can we find the value function by solving a system of linear equations?
I am following the book "Reinforcement Learning: An Introduction" by Richard Sutton and Andrew Barto, and they give an example of a problem for which the value function can be computed ...
- 143
0
votes
1
answer
465
views
What's the difference between a 1d tensor and a 2d tensor with 1 dimension?
I'm doing a TensorFlow tutorial, where they convert an array of the numbers [1,2,3] to a tensor like this:
...
- 167
-1
votes
1
answer
212
views
How can the gradient of the weight be calculated in the viewpoint of matrix calculus?
Let $\sigma(x)$ be sigmoid function. Consider the case where $\text{out}=\sigma(\vec{x} \times W + \vec{b})$, and we want to compute $\frac{\partial{\text{out}}}{\partial{w}
}.$
Set the dimension as ...
- 99
6
votes
1
answer
1k
views
How to express a fully connected neural network succintly using linear algebra?
I'm currently reading the paper Federated Learning with Matched Averaging (2020), where the authors claim:
A basic fully connected (FC) NN can be formulated as: $\hat{y} = \sigma(xW_1)W_2$ [...]
...
- 73
0
votes
1
answer
305
views
Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?
If the derivative is supposed to give the rate of change of a function at that point, then why is the derivative of the softmax layer (a vector) the Jacobian matrix, which has a different shape than ...
- 115
2
votes
0
answers
29
views
How to interpret the variance calculation in a Guassian process
I answered another question here about the mean prediction of a GP, but I have a hard time coming up with an intuitive explanation of the variance prediction of a GP.
Thew specific equation that I am ...
- 139
2
votes
1
answer
143
views
What exactly is the eigenspace of a graph (in spectral clustering)?
When we find the eigenvectors of a graph (say in the context of spectral clustering), what exactly is the vector space involved here? Of what vector space (or eigenspace) are we finding the ...
1
vote
0
answers
166
views
Is the 3d convolution associative given that it can be represented as matrix multiplication?
I'm trying to understand if a 3D convolution of the sort performed in a convolutional layer of a CNN is associative. Specifically, is the following true:
$$
X \otimes(W \cdot Q)=(X \otimes W) \cdot Q,
...
- 11
1
vote
0
answers
30
views
How to find distance between 2 points when dimensions are all of different nature?
I have a dataset with four features:
the x coordinate
the y coordinate
the velocity magnitude
angle
Now, I want to measure the distance between two points in the dataset, taking into account the ...
2
votes
2
answers
137
views
How does PCA work when we reduce the original space to 2 or higher-dimensional space?
How does PCA work when we reduce the original space to a 2 or higher-dimensional space? I understand the case when we reduce the dimensionality to $1$, but not this case.
$$\begin{array}{ll} \text{...
- 21
1
vote
0
answers
35
views
Human intuition behind SVD in case of recommendation system
This does not answer my question. I struggled very hard to understand the SVD from a linear-algebra point of view. But in some cases I failed to connect the dots. So, I started to see all the ...
- 111
2
votes
1
answer
247
views
What do we mean by 'principal angle between subspaces'?
I came across the term 'principal angle between subspaces' as a tool for comparing objects in images. All material that I found on the internet seems to deal with this idea in a highly mathematical ...
1
vote
0
answers
54
views
Simplifying Log Loss
I am reading through a paper (https://www.mitpressjournals.org/doi/pdf/10.1162/0891201053630273) where they describe logloss as a ranking function and can be simplified to the margin of the training ...
- 227
3
votes
1
answer
5k
views
How do you find the homography matrix given 4 points in both images?
I want to understand the process of finding a homography matrix given 4 points in both images. I am able to do that in python OpenCV, but I wonder how it works behind the scenes.
Suppose I have ...
- 175
1
vote
1
answer
68
views
Is there any way to apply linear transformations on a vector other than matrix multiplication?
I am trying to optimize the cost function calculation in regression analysis using a non-matrix multiplication based approach.
More specifically, I have a point $x = (1, 1, 2, 3)$, to which I want to ...
- 111
4
votes
1
answer
140
views
Why MLP cannot approximate a closed shape function?
[TL;DR]
I generated two classes Red and Blue on a 2D space. Red are points on Unit Circle and Blue are points on a Circle Ring with radius limits (3,4). I tried to train a Multi Layer Perceptron ...
- 143
1
vote
0
answers
35
views
CSP Formulation of an algebraic problem
Is anyone able to explain how to do this? I'm not looking for the complete answer, I would settle for a "how to for dummies" explanation of how this is supposed to be solved.
I understand ...
- 11
5
votes
2
answers
199
views
Which linear algebra book should I read to understand vectorized operations?
I am reading Goodfellow's book about neural networks, but I am stuck in the mathematical calculus of the back-propagation algorithm. I understood the principle, and some Youtube videos explaining this ...
- 151
1
vote
0
answers
34
views
Estimating Baselines using ALS
I am trying to figure out how ALS works when minimizing the following formula:
$\\ \\$
$\text{min}_{\lbrace b_u,b_i \rbrace} \sum_{(u,i)\in \mathcal{K}} (r_{ui} - \bar{r} - b_u - b_i )^2 + \lambda_{...
- 155
6
votes
1
answer
257
views
What does it mean to do multi-dimensional processing with tensors in tensor cores?
In some tweets about NeurIPS 2018, this video from NVIDIA appeared. At around 0.37, she says:
If you think about the current computations in our deep learning systems, they are all based on Linear ...
- 161
1
vote
1
answer
334
views
Using reinforcement learning to find a preconditioner for linear systems of the form Ax = b
Sparse linear systems are normally solved by using solvers like MINRES, Conjugate gradient, GMRES.
Efficient preconditioning, i.e., finding a matrix P such that PAx = Pb is easier to solve than the ...
- 31
2
votes
1
answer
808
views
Does k consistency always imply (k - 1) consistency?
From Russell-Norvig:
A CSP is strongly k-consistent if it is k-consistent and is also (k − 1)-consistent, (k − 2)-consistent, . . . all the way down to 1-consistent.
How can a CSP be k-consistent ...
- 131
5
votes
3
answers
551
views
How does neural network classifier classify from just drawing a decision plane?
I understand that a neural network basically distorts(non-linear transformation) and changes the perspective(linear transformations) of input space to draw a plane to classify data. How does the ...
- 326