For questions related to the gradient, a way of packing together all the partial derivative information of a function

51 questions
Filter by
Sorted by
Tagged with
8 views

### Find the pixel sensitivity of a Convolutional Neural Network

So assume I have a simple convolutional neural network with 2 or 3 convolutional layers, a pooling layer, and whatever else is typical. The network takes in an image and tries to produce an image that ...
• 1
19 views

• 11
4k views

### What does it mean by "gradient flow" in the context of neural networks?

Several research papers and textbooks (e.g. this) contain the phrase "gradient flow" in the context of neural networks. I am confused about whether it has any rigorous and formal way of ...
• 3,880
1k views

### What specifically is the gradient of the log of the probability in policy gradient methods?

I am getting tripped up slightly by how specifically the gradient is calculated in policy gradient methods (just the intuitive understanding of it). This Math Stack Exchange post is close, but I'm ...
• 161
1 vote
446 views

### What exactly do gradient-based saliency map tell us?

As far as I understand, gradients are supposed to tell us 1) the magnitude and 2) direction, to update a parameter such as to minimize the loss function. Regarding saliency maps, which use gradients ...
1 vote
80 views

### Rank of gradient-of-loss with respect to layer weights in an MLP

The paper: https://arxiv.org/abs/2110.11309, makes the following claim at the end of page 3: The gradient of loss $L$ with respect to weights $W_l$ of an MLP is a rank-1 matrix for each of B batch ...
• 121
44 views

### Which is more popular/common way of representing a gradient in AI community: as a row or column vector?

Consider the following remark about writing gradients from the chapter named Vector Calculus from the test book titled Mathematics for Machine Learning by Marc Peter Deisenroth et al. Remark (...
• 3,880
137 views

### What is the rigorous and formal definition for the direction pointed by a gradient?

Consider the following definition of derivative from the chapter named Vector Calculus from the test book titled Mathematics for Machine Learning by Marc Peter Deisenroth et al. Definition 5.2 (...
• 3,880
1 vote
297 views

### Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?

I am asking this question from the mathematical perspective of the vanishing and exploding gradient problems that we face generally during training deep neural networks. The chain rule of ...
• 3,880
236 views

### Isssue in understanding the derivation regarding mean squared error

The following derivation is taken from Chapter 5: Machine Learning Basics from the book titled Deep Learning (by Aaron Courville et al.) I am facing difficulty in understanding the zero derivative ...
• 3,880
340 views

### How many directions of gradients exist for a function in higher dimensional space?

Gradients are used in optimization algorithms. Based on the values of gradients, we generally update the weights of a neural network. It is known that gradients have a direction and the direction ...
• 3,880
1 vote
83 views

### What all does the gradient tells us other than the direction to move parameters?

Gradients are used in optimization algorithms. I know that a gradient gives us information about the direction in which one needs to update the weights of a neural network. We need to travel in the ...
• 3,880
1 vote
41 views

### What is the high-level algorithm followed by contemporary packages for the calculation of gradient?

Most of the neural network models in contemporary deep learning packages are trained based on gradients. Let $f: \mathbb{R}^m \rightarrow \mathbb{R}^n$ be a function for which we want to find a ...
• 3,880
1 vote
71 views

### Is there any significance for higher order gradients in artificial intelligence?

Although I don't know in detail, I am aware of the following facts regarding the use of gradients in some domains of artificial intelligence, especially in minimizing the training of neural networks. ...
• 3,880
270 views

### What does it mean by strong or sufficient gradient for training in this context?

It has been mentioned in the research paper titled Generative Adversarial Nets that generator need to maximize the function $\log D(G(z))$ instead of minimizing $\log(1 −D(G(z)))$ since the former ...
• 3,880
150 views

### What is meant by "well-behaved gradient" in this context?

Consider the following statement (from the paper Generative Adversarial Nets) about the success of discriminative models So far, the most striking successes in deep learning have involved ...
• 3,880
812 views

### How to calculate the gradient penalty proposed in "Improved Training of Wasserstein GANs"?

The research paper titled Improved Training of Wasserstein GANs proposed a gradient penalty in order to avoid undesired behavior due to weight clipping of the discriminator. We now propose an ...
• 3,880
1 vote
134 views

### What does it mean by "zeros the networks parameters gradients" in the context of training a neural network?

Consider the following PyTorch code ...
• 3,880
1 vote