Questions tagged [derivative]
The derivative tag has no usage guidance.
20
questions
1
vote
0
answers
7
views
Neural Networks that fit vector transforms
I have a CNN that is image to image and maps a binary image input to a binary image output. These are usually simple shapes, like a rectangle or a circle. Usually they become smoothed a bit (the ...
- 11
0
votes
0
answers
60
views
Why does "in-place" mutation cause automatic differentiation to fail, and how to write code to avoid this problem
I work with a few different automatic differentiation frameworks, including pytorch, Jax, and Flux in Julia. Periodically I run some code and I get errors about mutations or operations occurring "...
- 207
1
vote
2
answers
96
views
Direct formula for calculating the optimum matrix which minimizes the perceptron error
Suppose we have a perceptron without bias and $f(x) = x$ as activation function and matrices $X,Y,W$ that input training data are columns of matrix $X$, $Y$ is targets matrix (columns are ordered with ...
- 113
0
votes
0
answers
20
views
In MLP, Why in the formula to calculate delta, the derivation of the cost function "(predicted - expected)" just as "(predicted - expected)" itself?
I know that we have to perform the derivation of the terms, using the chain rule, to obtain the equations to calculate the gradients of the neurons.
In this article: https://machinelearningmastery.com/...
- 267
1
vote
0
answers
40
views
In MLP, to calculate the delta, do I need to calculate the derivative of the cost function? Or can I just use the cost function result?
In Multi Layer Perceptron networks, I have a question: in the formula to calculate the error in the output layer, Some articles say it is like this ...
- 267
1
vote
0
answers
38
views
Neural network learns to mimic distribution of classes in dataset instead of using signal from input
I'm trying to implement example from a classic AI paper named "Learning representations by back-propagating errors" by Hinton et al.
Example aims at training network able to predict third ...
- 11
3
votes
1
answer
644
views
How is the max function differentiable wrt multiple arguments?
I recently came across an answer on StackOverflow that mentioned the max function being differentiable with respect to its values.
From my current understanding of ...
- 584
0
votes
0
answers
95
views
Why Is There The Term 1/m In Backpropagation
In backpropagation the gradients are used to update the weights using the formula
$$w = w - \alpha \frac{dL}{dw}$$
and the loss gradient w.r.t. weights is
$$\frac{dL}{dw} = \frac{dL}{dz} \frac{dz}{dw} ...
- 31
0
votes
1
answer
87
views
What is the correct partial derivative of $Y^c$ with respect to $A_{ij}^{kc}$?
I have a question about the Grad-CAM++ paper. I do not understand how the following equation (10) for the alphas is obtained:
$$
\alpha_{ij}^{kc} =
\frac{\frac{\partial^2 Y^c}{(\partial A_{ij}^k)^2}}
{...
- 141
0
votes
2
answers
211
views
How the vector-space isomorphism between $\mathbb{R}^{m \times n}$ and $\mathbb{R}^{mn}$ guarantees reshaping matrices to vectors?
Consider the following paragraph from section 5.4 Gradients fo Matrices of the chapter Vector Calculus from the textbook titled Mathematics for Machine Learning by Marc Peter Deisenroth et al.
Since ...
- 3,950
0
votes
1
answer
142
views
What is the rigorous and formal definition for the direction pointed by a gradient?
Consider the following definition of derivative from the chapter named Vector Calculus from the test book titled Mathematics for Machine Learning by Marc Peter Deisenroth et al.
Definition 5.2 (...
- 3,950
0
votes
1
answer
64
views
How to understand slope of a (non-convex) function at a point in domain?
Consider the following paragraph from Numerical Computation of deep learning book that says derivative as a slope of the function curve at a point
Suppose we have a function $y= f(x)$, where both $x$ ...
- 3,950
1
vote
2
answers
914
views
What does it mean "having Lipschitz continuous derivatives"?
We can enforce some constraints on functions used in deep learning in order to guarantee optimizations. You can find it in Numerical Computation of the deep learning book.
In the context of deep ...
- 3,950
2
votes
2
answers
249
views
Why does critical points and stationary points are used interchangeably?
Consider the following paragraph from Numerical Computation of the deep learning book.
When $f'(x) = 0$, the derivative provides no information about which
direction to move. Points where $f'(x)$ = 0 ...
- 3,950
0
votes
2
answers
38
views
Reason for relaxing limit in derivative in this context?
Consider the following paragraph from NUMERICAL COMPUTATION of the deep learning book..
Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real
numbers. The derivative of this function ...
- 3,950
1
vote
0
answers
82
views
BlackOut - ICLR 2016: need help understanding the cost function derivative
In the ICLR 2016 paper BlackOut: Speeding up Recurrent Neural Network Language Models with very Large Vocabularies, on page 3, for eq. 4:
$$ J_{ml}^s(\theta) = log \ p_{\theta}(w_i | s) $$
They have ...
- 151
2
votes
0
answers
70
views
What is the dimensionality of these derivatives in the paper "Active Learning for Reward Estimation in Inverse Reinforcement Learning"?
I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning.
I'm specifically referring to section 2.3 of the paper.
Let's ...
- 141
0
votes
0
answers
354
views
Derivation of regularized cost function w.r.t activation and bias
In regularzied cost function a L2 regularization cost has been added.
Here we have already calculated cross entropy cost w.r.t $A, W$.
As mentioned in the regularization notebook (see below) in ...
- 151
4
votes
1
answer
318
views
Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?
I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation.
As per my knowledge, ...
- 151
0
votes
1
answer
170
views
Backpropagation: Chain Rule to the Third Last Layer
I'm trying to solve dLoss/dW1. The network is as in picture below with identity activation at all neurons:
Solving dLoss/dW7 is simple as there's only 1 way to output:
$Delta = Out-Y$
$Loss = abs(...
- 1,293