Questions tagged [derivative]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
4 votes
1 answer
204 views

Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?

I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation. As per my knowledge, ...
user avatar
  • 151
2 votes
2 answers
119 views

Why does critical points and stationary points are used interchangeably?

Consider the following paragraph from Numerical Computation of the deep learning book. When $f'(x) = 0$, the derivative provides no information about which direction to move. Points where $f'(x)$ = 0 ...
user avatar
  • 3,201
2 votes
0 answers
63 views

What is the dimensionality of these derivatives in the paper "Active Learning for Reward Estimation in Inverse Reinforcement Learning"?

I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning. I'm specifically referring to section 2.3 of the paper. Let's ...
user avatar
1 vote
1 answer
80 views

What does it mean "having Lipschitz continuous derivatives"?

We can enforce some constraints on functions used in deep learning in order to guarantee optimizations. You can find it in Numerical Computation of the deep learning book. In the context of deep ...
user avatar
  • 3,201
1 vote
0 answers
72 views

BlackOut - ICLR 2016: need help understanding the cost function derivative

In the ICLR 2016 paper BlackOut: Speeding up Recurrent Neural Network Language Models with very Large Vocabularies, on page 3, for eq. 4: $$ J_{ml}^s(\theta) = log \ p_{\theta}(w_i | s) $$ They have ...
user avatar
  • 151
0 votes
2 answers
36 views

Reason for relaxing limit in derivative in this context?

Consider the following paragraph from NUMERICAL COMPUTATION of the deep learning book.. Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real numbers. The derivative of this function ...
user avatar
  • 3,201
0 votes
0 answers
25 views

Why Is There The Term 1/m In Backpropagation

In backpropagation the gradients are used to update the weights using the formula $$w = w - \alpha \frac{dL}{dw}$$ and the loss gradient w.r.t. weights is $$\frac{dL}{dw} = \frac{dL}{dz} \frac{dz}{dw} ...
user avatar
0 votes
1 answer
58 views

What is the correct partial derivative of $Y^c$ with respect to $A_{ij}^{kc}$?

I have a question about the Grad-CAM++ paper. I do not understand how the following equation (10) for the alphas is obtained: $$ \alpha_{ij}^{kc} = \frac{\frac{\partial^2 Y^c}{(\partial A_{ij}^k)^2}} {...
user avatar
  • 131
0 votes
0 answers
28 views

How to calculate the gradient (or derivative) of y = f(x) of y w.r.t x where y represents the order statistics divided by median of x?

How to calculate the gradient (or derivative) of y = f(x) of y w.r.t x where y represents the order statistics divided by median of x? For instance x is ...
user avatar
  • 101
0 votes
1 answer
58 views

How the vector-space isomorphism between $\mathbb{R}^{m \times n}$ and $\mathbb{R}^{mn}$ guarantees reshaping matrices to vectors?

Consider the following paragraph from section 5.4 Gradients fo Matrices of the chapter Vector Calculus from the textbook titled Mathematics for Machine Learning by Marc Peter Deisenroth et al. Since ...
user avatar
  • 3,201
0 votes
1 answer
82 views

What is the rigorous and formal definition for the direction pointed by a gradient?

Consider the following definition of derivative from the chapter named Vector Calculus from the test book titled Mathematics for Machine Learning by Marc Peter Deisenroth et al. Definition 5.2 (...
user avatar
  • 3,201
0 votes
1 answer
39 views

How to understand slope of a (non-convex) function at a point in domain?

Consider the following paragraph from Numerical Computation of deep learning book that says derivative as a slope of the function curve at a point Suppose we have a function $y= f(x)$, where both $x$ ...
user avatar
  • 3,201
0 votes
0 answers
137 views

Derivation of regularized cost function w.r.t activation and bias

In regularzied cost function a L2 regularization cost has been added. Here we have already calculated cross entropy cost w.r.t $A, W$. As mentioned in the regularization notebook (see below) in ...
user avatar
  • 151
0 votes
1 answer
65 views

Backpropagation: Chain Rule to the Third Last Layer

I'm trying to solve dLoss/dW1. The network is as in picture below with identity activation at all neurons: Solving dLoss/dW7 is simple as there's only 1 way to output: $Delta = Out-Y$ $Loss = abs(...
user avatar
  • 1,213