8
votes
Accepted
How to express a fully connected neural network succintly using linear algebra?
The equation $$\hat{y} = \sigma(xW_\color{green}{1})W_\color{blue}{2} \tag{1}\label{1}$$ is the equation of the forward pass of a single-hidden layer fully connected and feedforward neural network, i....
- 38.2k
7
votes
Accepted
How do you find the homography matrix given 4 points in both images?
To understand homographies and how to find them, you will need a good dose of projective geometry. I will briefly describe some preliminary concepts that you need to know before trying to find the ...
- 38.2k
4
votes
Is there any difference between affine transformation and linear transformation?
In linear algebra, a linear transformation (aka linear map or linear transform) $f: \mathcal{V} \rightarrow \mathcal{W}$ is a function that satisfies the following two conditions
$f(u + v)=f(u)+f(v)$ ...
- 38.2k
3
votes
How can we find the value function by solving a system of linear equations?
Provided you have a finite number of states and actions, then there will not be an infinite number of terms. Therefore the state and action spaces need to be discrete and finite before the quote from ...
- 27k
3
votes
Accepted
Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?
When you use the softmax activation function is usually as a last layer of your network and to get an output that is a vector. Now your confusion is about shapes, so let's review a bit of calculus.
If ...
- 278
3
votes
Accepted
Why MLP cannot approximate a closed shape function?
In neural networks, the family of functions and the shapes that they can make for decision surfaces is determined by the activation function you use (in your case, ...
- 27k
3
votes
Accepted
Which linear algebra book should I read to understand vectorized operations?
If you already have two years of a bachelor's of mathematics, I recommend part I of the book that you're mentioning. That part of the book reviews the main mathematics used in the optimization of ...
2
votes
Does k consistency always imply (k - 1) consistency?
Define P as a CSP where X, Y are the variables, domain of both is {1,2,3,4} and conditions in normal form are:
node-condition X<4
arc-condition X=Y
P is 2-consistent (arc consistent) because for ...
- 1,282
2
votes
What does it mean to do multi-dimensional processing with tensors in tensor cores?
Both PR and Theoretically Valid
Although Anima Anandkumar's presentation is a puff piece for NVidia, her representation is not contrary to theory.
... next level [above NVidia's GPU success] ... ...
- 7,393
2
votes
How does PCA work when we reduce the original space to 2 or higher-dimensional space?
You might want to have a look at the wikipedia article of PCA, where it says:
"The $k$th component can be found by subtracting the first $k − 1$ principal components from $\mathbf{X}$:"
$$\...
- 620
2
votes
Is there any difference between affine transformation and linear transformation?
The fact is you can always express an affine transformation as a linear transformation (more convenient because it is just a matrix/dot product).
For instance, given an input $\textbf{x}=[x_1, ..., ...
- 21
2
votes
How does neural network classifier classify from just drawing a decision plane?
If you are working with supervised learning, each training example has a label. That label is your classification of the provided input. Just like linear or logistic regression, if your problem only ...
- 121
1
vote
How Does The Scaled Dot Product's Dimensions Work Out In Mult-Head Attention?
Say you have 3-dimensional tensors Q & K of shape (B, T, C) (for Batch, Time, and Channels).
K is transposed. 'Transposing' a 3-d tensor in this case means ...
- 1,574
1
vote
Encoding data in which the order of features does not matter
Is there maybe a better way [to combine them into one vector in such a way that their order does not matter]?
There are many ways to perform feature fusion (see ref). The way you described is one way ...
- 882
1
vote
Does $(\langle w, x \rangle + b) = ||x - x'||$ hold?
I think that your missing the fact that $r$ is not a random constant, but precisely the minimum distance we want to ensure between each point and the separation hyperplane (called margin). And indeed ...
- 4,948
1
vote
Do solving system of linear equations required anywhere in contemporarty deep learning?
I guess a first distinction should be made between deep learning as a whole or deep learning as architecture.
I think the paragraph you quote refers to solving systems of linear equations as a simple ...
- 4,948
1
vote
Accepted
How can we find the value function by solving a system of linear equations?
First of all, we assume that we have a finite MDP, i.e. the set of states $\mathcal{S}$, the set of actions $\mathcal{A}$ and the set of rewards $\mathcal{R}$ all have a finite number of elements (I ...
- 38.2k
1
vote
Accepted
What's the difference between a 1d tensor and a 2d tensor with 1 dimension?
The required shape of the tensor $T$ depends on the shape of other tensors that are involved in the same operations of that same tensor $T$ and the required/desired shape of the resulting tensor, in ...
- 38.2k
1
vote
Accepted
What exactly is the eigenspace of a graph (in spectral clustering)?
In spectral clustering we not find the eigenvectors of a graph (a graph is not a matrix) but the eigenvalues/eigenvectors of the Laplacian matrix related to the adjacency matrix of the graph:
graph =&...
- 1,282
1
vote
What do we mean by 'principal angle between subspaces'?
Let's consider the case where you have two photos, one base photo and one other photo which is a scaled version of the base photo. Consider then that you could create a 'mapping' from the base photo ...
- 176
1
vote
Is there any way to apply linear transformations on a vector other than matrix multiplication?
You can sometimes exploit the structure of your matrix to perform faster matrix multiplication. For example, if your matrix is sparse (or dense), there are algorithms that exploit this fact.
In your ...
- 38.2k
1
vote
Which linear algebra book should I read to understand vectorized operations?
Linear Algebra Done Right by Axler seems to be the best book on linear algebra, with a brisk and modern approach.
- 2,051
1
vote
Using reinforcement learning to find a preconditioner for linear systems of the form Ax = b
Reinforcement Learning is a method for learning to perform beneficial actions in an environment. One way this is accomplished is by learning to predict useful actions as a function of the observed ...
- 9,097
1
vote
How does neural network classifier classify from just drawing a decision plane?
For other who were wondering the same questions as me, I'll answer it.
My view above was inconsistent. Ultimately the last layer of simple feed-forward networks don't have any special properties ...
- 326
Only top scored, non community-wiki answers of a minimum length are eligible