Skip to main content

Questions tagged [linear-algebra]

For questions about the use/aspects/implementation/intuition/mathematical proofs of various Linear Algebra methods used in Machine Learning and AI algorithms.

Filter by
Sorted by
Tagged with
0 votes
0 answers
21 views

Analyzing a deep learning model by constructing a Matrix from input and output data

I am currently completing my bachelor’s thesis, and recently my supervisor suggested that I could strengthen my arguments by explaining why deep learning models perform so well in my case. To give you ...
glaand's user avatar
  • 101
0 votes
1 answer
40 views

Why are the Q and K matrices two separate matrices in attention?

If I understand correctly the attention layer is represented as $$ \begin{align} &softmax(\frac{Q K^T}{\sqrt{d_k}}) V \\ = &softmax(\frac{(s W_q) (s W_k)^T}{\sqrt{d_k}}) V \\ = &softmax(\...
fakedrake's user avatar
  • 103
0 votes
1 answer
56 views

Is it possible to simplify max(max(a,b),c)?

Like the title says, is there any way to simplify the piece wise linear function of max(max(a,b),c) into some linear combination of max(a,b), max(b,c), max(a,c)?
Xiaoyang Li's user avatar
1 vote
2 answers
138 views

How are sentences turned into a vector in LLM

My understanding of Large Language Models like GPT is that they are special kinds of deep neural networks specifically trained to predict the next word, given the beginning of a sentence. I understand ...
Weier's user avatar
  • 131
1 vote
1 answer
71 views

Is the multi-headed projection matrix in self-attention redundant?

As I understand it, the forward pass for a transformer model looks as follows: x += self_attention(x) x = layernorm(x) x += ffn(x) Breaking that down a bit (excuse ...
Sue Doh Nimh's user avatar
0 votes
1 answer
62 views

What are the formulas for the MAE , MSE when output is a vector?

What are the formulas for the MAE , MSE when output is a vector? MAE: Mean Absolute Error MSE: Mean squared error
DSPinfinity's user avatar
0 votes
0 answers
15 views

Linear Discriminant Analysis on a transformed space

Let $S$ be a finite subset of a $\mathbb{R}^k$ partitioned into $N$ subsets $S_1, \ldots, S_N$ and let $n_j = |S_j|$. The between-groups sum of squares of the partition is defined as $$bSS(S_1,\ldots, ...
Alberto's user avatar
  • 101
1 vote
2 answers
178 views

Why my best fit line is not having a single straight line | Multiple Linear Regression

I am working on Multiple Linear Regression (Multiple variables). I am been able to predict and get a good r2 score. But I am not sure that I understood the part of plotting the best fit line, I can't ...
Niranjanadas M's user avatar
1 vote
0 answers
13 views

Comparing Auto-regressive Encoder-Decoders and Topological Neural Networks

I am interested in what insights can be gained about the mathematical class of auto-regressive encoder-decoders (LLMs), by comparing them to topological neural networks. Specifically, I am looking for ...
Julius Hamilton's user avatar
3 votes
0 answers
497 views

How to work with multiple embeddings?

This is a conceptual gap that I have concerning embeddings, and would really appreciate some help closing it. I understand when you embed a corpus for, let's say, a question-and-answer task you can ...
Ian Murray's user avatar
2 votes
0 answers
78 views

Is orthogonal initialization still useful when hidden layer sizes vary?

Pytorch's orthogonal initialization cites "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks ", Saxe, A. et al. (2013), which gives as reason for the ...
Gabi's user avatar
  • 121
0 votes
1 answer
47 views

Shape of biases in Transformer's Feedforward Network

In transformer network (Vaswani et al., 2017), the feedforward networks have equation: $$\mathrm{FNN}(x) = \max(0, xW_1 + b_1) W_2 + b_2$$ where $x \in \mathbb{R}^{n \times d_\mathrm{model}}$, $W_1 \...
poglhar's user avatar
  • 23
0 votes
1 answer
228 views

How Does The Scaled Dot Product's Dimensions Work Out In Mult-Head Attention?

I don't understand how self-attention works with batched values for the $Q \times K^T $ step. According to the diagram below (assume 1 head), once we get past the first 3 linear steps, we arrive at ...
rkuang25's user avatar
0 votes
1 answer
34 views

Is $i$ indexing the first or second dimension in $\mathbf{x}_i$, where $\mathbf{x} \in \mathbb{R}^{n\times d}$?

I was reading the following notes on the math behind transformers and was confused about what $\mathbf{x}_i$ is? If $\mathbf{x} \in \mathbb{R}^{n\times d}$, then is the $i$ indexing the $n$ or the the ...
play's user avatar
  • 133
1 vote
2 answers
370 views

Encoding data in which the order of features does not matter

My input to the model is a set of features that I encode in the form of five vectors of the same size consisting only of 0 and 1. I now want to combine them into one vector in such a way that their ...
Kasia's user avatar
  • 303
1 vote
1 answer
167 views

How does using complex weights in a neural network affect performance?

If you switch a neural network from real weights to complex weights, you're roughly doubling the size of the network, and increasing the computational load by a factor of 2 to 4. My question is, in ...
chausies's user avatar
  • 150
0 votes
1 answer
67 views

Does $(\langle w, x \rangle + b) = ||x - x'||$ hold?

Currently, I am trying to understand the mathematics of SVM's using the textbook 'Mathematics for Machine Learning' by Deisenrot et. al. On page 375, they define the distance between a point $x$ and ...
Philipp Kunz's user avatar
1 vote
0 answers
174 views

Where do the characteristics of self-attention come into play in Linformer's proof that self-attention is low rank?

In Linformer's proof that self-attention is low rank in their paper, I don't see how it doesn't generalize to every matrix. They don't utilize any specifics of self-attention (the entire proof feels ...
mshlis's user avatar
  • 2,369
1 vote
1 answer
75 views

Why does $I_N + D^{-\frac{1}{2}}AD^{-\frac{1}{2}}$ have eigenvalues in the range [0, 2]?

In Semi-supervised classification with Graph Convolutional Networks, I am unable to understand a few things. Given an undirected graph having adjacency matrix $A$, degree matrix $D_{ii} = \sum_j A_{...
willtryagain's user avatar
2 votes
2 answers
536 views

Do solving system of linear equations required anywhere in contemporarty deep learning?

Consider the following from Numerical Computation chapter of Deep Learning book Machine learning algorithms usually require a high amount of numerical computation. This typically refers to algorithms ...
hanugm's user avatar
  • 3,890
3 votes
2 answers
2k views

Is there any difference between affine transformation and linear transformation?

Consider the following statements from A Simple Custom Module of PyTorch's documentation To get started, let’s look at a simpler, custom version of PyTorch’s Linear module. This module applies an ...
hanugm's user avatar
  • 3,890
3 votes
2 answers
1k views

How can we find the value function by solving a system of linear equations?

I am following the book "Reinforcement Learning: An Introduction" by Richard Sutton and Andrew Barto, and they give an example of a problem for which the value function can be computed ...
phil's user avatar
  • 143
0 votes
1 answer
546 views

What's the difference between a 1d tensor and a 2d tensor with 1 dimension?

I'm doing a TensorFlow tutorial, where they convert an array of the numbers [1,2,3] to a tensor like this: ...
Kokodoko's user avatar
  • 167
-1 votes
1 answer
248 views

How can the gradient of the weight be calculated in the viewpoint of matrix calculus?

Let $\sigma(x)$ be sigmoid function. Consider the case where $\text{out}=\sigma(\vec{x} \times W + \vec{b})$, and we want to compute $\frac{\partial{\text{out}}}{\partial{w} }.$ Set the dimension as ...
최선웅's user avatar
6 votes
1 answer
2k views

How to express a fully connected neural network succintly using linear algebra?

I'm currently reading the paper Federated Learning with Matched Averaging (2020), where the authors claim: A basic fully connected (FC) NN can be formulated as: $\hat{y} = \sigma(xW_1)W_2$ [...] ...
user1360448's user avatar
0 votes
1 answer
375 views

Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

If the derivative is supposed to give the rate of change of a function at that point, then why is the derivative of the softmax layer (a vector) the Jacobian matrix, which has a different shape than ...
Ilknur Mustafa's user avatar
2 votes
0 answers
32 views

How to interpret the variance calculation in a Guassian process

I answered another question here about the mean prediction of a GP, but I have a hard time coming up with an intuitive explanation of the variance prediction of a GP. Thew specific equation that I am ...
Joff's user avatar
  • 139
2 votes
1 answer
154 views

What exactly is the eigenspace of a graph (in spectral clustering)?

When we find the eigenvectors of a graph (say in the context of spectral clustering), what exactly is the vector space involved here? Of what vector space (or eigenspace) are we finding the ...
Manish Kausik Hari Baskar's user avatar
1 vote
0 answers
179 views

Is the 3d convolution associative given that it can be represented as matrix multiplication?

I'm trying to understand if a 3D convolution of the sort performed in a convolutional layer of a CNN is associative. Specifically, is the following true: $$ X \otimes(W \cdot Q)=(X \otimes W) \cdot Q, ...
HereItIs's user avatar
1 vote
0 answers
30 views

How to find distance between 2 points when dimensions are all of different nature?

I have a dataset with four features: the x coordinate the y coordinate the velocity magnitude angle Now, I want to measure the distance between two points in the dataset, taking into account the ...
Manish Kausik Hari Baskar's user avatar
2 votes
2 answers
140 views

How does PCA work when we reduce the original space to 2 or higher-dimensional space?

How does PCA work when we reduce the original space to a 2 or higher-dimensional space? I understand the case when we reduce the dimensionality to $1$, but not this case. $$\begin{array}{ll} \text{...
VN Pikachu's user avatar
1 vote
0 answers
41 views

Human intuition behind SVD in case of recommendation system

This does not answer my question. I struggled very hard to understand the SVD from a linear-algebra point of view. But in some cases I failed to connect the dots. So, I started to see all the ...
F.C. Akhi's user avatar
  • 111
2 votes
1 answer
285 views

What do we mean by 'principal angle between subspaces'?

I came across the term 'principal angle between subspaces' as a tool for comparing objects in images. All material that I found on the internet seems to deal with this idea in a highly mathematical ...
Manish Kausik Hari Baskar's user avatar
1 vote
0 answers
59 views

Simplifying Log Loss

I am reading through a paper (https://www.mitpressjournals.org/doi/pdf/10.1162/0891201053630273) where they describe logloss as a ranking function and can be simplified to the margin of the training ...
Jacob B's user avatar
  • 247
4 votes
1 answer
6k views

How do you find the homography matrix given 4 points in both images?

I want to understand the process of finding a homography matrix given 4 points in both images. I am able to do that in python OpenCV, but I wonder how it works behind the scenes. Suppose I have ...
Hissaan's user avatar
  • 185
1 vote
1 answer
70 views

Is there any way to apply linear transformations on a vector other than matrix multiplication?

I am trying to optimize the cost function calculation in regression analysis using a non-matrix multiplication based approach. More specifically, I have a point $x = (1, 1, 2, 3)$, to which I want to ...
Akshay Tilekar's user avatar
4 votes
1 answer
182 views

Why MLP cannot approximate a closed shape function?

[TL;DR] I generated two classes Red and Blue on a 2D space. Red are points on Unit Circle and Blue are points on a Circle Ring with radius limits (3,4). I tried to train a Multi Layer Perceptron ...
entropyfeverone's user avatar
1 vote
0 answers
35 views

CSP Formulation of an algebraic problem

Is anyone able to explain how to do this? I'm not looking for the complete answer, I would settle for a "how to for dummies" explanation of how this is supposed to be solved. I understand ...
Steve's user avatar
  • 11
5 votes
2 answers
221 views

Which linear algebra book should I read to understand vectorized operations?

I am reading Goodfellow's book about neural networks, but I am stuck in the mathematical calculus of the back-propagation algorithm. I understood the principle, and some Youtube videos explaining this ...
lolveley's user avatar
  • 151
1 vote
0 answers
36 views

Estimating Baselines using ALS

I am trying to figure out how ALS works when minimizing the following formula: $\\ \\$ $\text{min}_{\lbrace b_u,b_i \rbrace} \sum_{(u,i)\in \mathcal{K}} (r_{ui} - \bar{r} - b_u - b_i )^2 + \lambda_{...
NaveganTeX's user avatar
6 votes
1 answer
269 views

What does it mean to do multi-dimensional processing with tensors in tensor cores?

In some tweets about NeurIPS 2018, this video from NVIDIA appeared. At around 0.37, she says: If you think about the current computations in our deep learning systems, they are all based on Linear ...
wrong_path's user avatar
1 vote
1 answer
351 views

Using reinforcement learning to find a preconditioner for linear systems of the form Ax = b

Sparse linear systems are normally solved by using solvers like MINRES, Conjugate gradient, GMRES. Efficient preconditioning, i.e., finding a matrix P such that PAx = Pb is easier to solve than the ...
timudk's user avatar
  • 31
2 votes
1 answer
897 views

Does k consistency always imply (k - 1) consistency?

From Russell-Norvig: A CSP is strongly k-consistent if it is k-consistent and is also (k − 1)-consistent, (k − 2)-consistent, . . . all the way down to 1-consistent. How can a CSP be k-consistent ...
amad-person's user avatar
5 votes
3 answers
579 views

How does neural network classifier classify from just drawing a decision plane?

I understand that a neural network basically distorts(non-linear transformation) and changes the perspective(linear transformations) of input space to draw a plane to classify data. How does the ...
Daniel's user avatar
  • 326