14

Neural networks are essentially just repeated matrix multiplications and applications of an activation function, so you really don't need a great deal of linear algebra to construct a simple neural network — if you understand how to multiply matrices, that's probably sufficient. The harder bit is the training process which is typically done through ...


3

If you already have two years of a bachelor's of mathematics, I recommend part I of the book that you're mentioning. That part of the book reviews the main mathematics used in the optimization of neural nets (in part 1), and then actually goes through the various models in detail in the later parts. The review is done at a level that is suitable for someone ...


2

The adjective bad isn't mathematically descriptive. A better term is sub-optimal, which implies the state of learning might appear optimal based on current information but the optimal solution from among all possibilities is not yet located. Consider a graph representing a loss function, one of the names to measure disparity between the current learning ...


2

Formally speaking $x_6$ is a function of $w_{16},\ w_{26}$ and $w_{36}$, that is $$x_6 =f(w_{16}, w_{26}, w_{36})=w_{16}y_1 + w_{26}y_2 + w_{36}y_3.$$ The derivative w.r.t. $w_{26}$ is $$\frac{\partial x_6}{\partial w_{26}}= \frac{\partial w_{16}y_1}{\partial w_{26}} +\frac{\partial w_{26}y_2}{\partial w_{26}} +\frac{\partial w_{36}y_3}{\partial w_{26}} = 0 ...


2

$F_l$ is the activation of the filter. They state in the paper that they base their method on VGG-Network, which uses ReLU as its activation function. In fact, VGG uses it in all of its hidden layers. ReLU is defined as $$f(x) = max(0,x)$$ Since ReLU is 0 for all x's below 0, the equation above holds; When x is non-positive, all terms in the loss function ...


1

Answer: Calculus James Stewart is the best for a beginner. I started to learn Calculus studying engineering with James Stewart Calculus ( maybe the best for beginners and is really didactic ), Problems in Mathematical Analysis Demidovich ( best for me because simplicity, fast, but few multivariable focus and difficult for learn ), Nikolai Piskunov - ...


1

To give some practical advice, it is important to understand parts of calculus. This is mainly because Backpropagation is a leaky abstraction in modern libraries. In a nutshell, there is a lot which can go wrong (exploding or vanishing gradient for example) and you will need knowledge about gradient descent to handle it. I highly recommend Andrej Karpathys ...


1

The identity $$\nabla_{\theta} P(\tau \mid \theta) = P(\tau \mid \theta) \nabla_{\theta} \log P(\tau \mid \theta)\tag{1}\label{1},$$ which can also be written as \begin{align} \nabla_{\theta} \log P(\tau \mid \theta) &= \frac{\nabla_{\theta} P(\tau \mid \theta)}{P(\tau \mid \theta)}\\ &=\frac{1}{P(\tau \mid \theta)} \nabla_{\theta} P(\tau \mid \...


1

Linear Algebra Done Right by Axler seems to be the best book on linear algebra, with a brisk and modern approach.


1

This is just an application of the chain rule. The same chapter has "Proof of the four fundamental equations" section, which proves BP1-2, while PB3-4 are left as exercise to the reader. I agree that it's a good exercise indeed, that's why I encourage you to stop here and try to prove it yourself using a chain rule. Now, if you decided to read further, here'...


Only top voted, non community-wiki answers of a minimum length are eligible