For questions related to the gradient, a way of packing together all the partial derivative information of a function

22 questions
Filter by
Sorted by
Tagged with
51 views

### PyTorch torch.no_grad vs torch.inference_mode [closed]

PyTorch has new functionality torch.inference_mode as of v1.9 which is "analogous to torch.no_grad... Code run under this ...
29 views

### Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?

I am asking this question from the mathematical perspective of the vanishing and exploding gradient problems that we face generally during training deep neural networks. The chain rule of ...
26 views

### Isssue in understanding the derivation regarding mean squared error

The following derivation is taken from Chapter 5: Machine Learning Basics from the book titled Deep Learning (by Aaron Courville et al.) I am facing difficulty in understanding the zero derivative ...
45 views

### How many directions of gradients exist for a function in higher dimensional space?

Gradient is used in optimization algorithms. Based on the values of gradient, we generally update weights of a neural network. It is known that gradient have a direction and the direction opposite to ...
25 views

### What all does the gradient tells us?

Gradients are used in optimization algorithms. I know that a gradient gives us information about the direction in which one needs to update the weights of a neural network. We need to travel in the ...
28 views

### What is the high-level algorithm followed by contemporary packages for the calculation of gradient?

Most of the neural network models in contemporary deep learning packages are trained based on gradients. Let $f: \mathbb{R}^m \rightarrow \mathbb{R}^n$ be a function for which we want to find a ...
25 views

### Is there any significance for higher order gradients in artificial intelligence?

Although I don't know in detail, I am aware of the following facts regarding the usefulness of gradients in some domains of artificial intelligence, especially in optimization. First order gradient: ...
45 views

### What does it mean by strong or sufficient gradient for training in this context?

It has been mentioned in the research paper titled Generative Adversarial Nets that generator need to maximize the function $\log D(G(z))$ instead of minimizing $\log(1 −D(G(z)))$ since the former ...
18 views

### How to handle critical points during generator training?

Using an MLP as generator introduces multiple critical points in parameter space. You can read this excerpt from research paper titled Generative Adversarial Nets In practice, adversarial nets ...
31 views

### What is meant by "well-behaved gradient" in this context?

Consider the following statement about the success of discriminative models So far, the most striking successes in deep learning have involved discriminative models, usually those that map a high-...
32 views

### How to calculate the gradient penalty proposed in "Improved Training of Wasserstein GANs"?

The research paper titled Improved Training of Wasserstein GANs proposed a gradient penalty in order to avoid undesired behavior due to weight clipping of the discriminator. We now propose an ...
35 views

### What are the input and output gradients in PyTorch?

Suppose I want to train a neural network with $m-$length inputs of form $x = [x_1, x_2, x_3, \cdots, x_m]$ and $n-$length outputs of form $y = [y_1, y_2, y_3, \cdots, y_n]$. Let the number of ...
41 views

### What does it mean by "zeros the networks parameters gradients" in the context of training a neural network?

Consider the following PyTorch code ...
Suppose we generate the vector output $y$ from model $h(x, \theta)$, with input $x$ and parameters $\theta$. Reverse mode differentiation says that we can calculate the gradient \begin{align*} \...