2022 Developer Survey is open! Take survey.

Questions tagged [gradient-descent]

For questions surrounding gradient descent, a method for finding the optimum state of a parameterized function based on another function often called the loss or error function. It iteratively descends the loss surface to the minimum loss by adjusting parameters based on the product of the partial derivatives comprising the gradient and a learning rate.

Filter by
Sorted by
Tagged with
0 votes
0 answers
3 views

At which point, does the momentum based GD helps really in this figure?

Classical gradient descent algorithms sometimes overshoot and escape minima as they depend on the gradient only. You can see such a problem during the update from point 6. In classical GD algorithm, ...
user avatar
  • 3,029
0 votes
0 answers
27 views

Resolving Derivation Discrepancies for Differentiating through Optimization Paths

I'm reading the paper "Optimizing Millions of Hyperparameters by Implicit Differentiation". The key contribution of the paper is to show that you can replace optimizing through the ...
user avatar
  • 1
0 votes
0 answers
34 views

How to explain near zero gradients on first epochs?

As I understand the gradient should reflect how near the weights are to the optimal values. In this way i will expect that on the first epochs the gradients far from zero or at least not mostly zero ...
user avatar
0 votes
0 answers
10 views

How to interpret integrated gradients in an NLP toxic text classification use-case?

I am trying to understand how integrated gradients work in the NLP case. Let $F: \mathbb{R}^{n} \rightarrow[0,1]$ a function representing a neural network, $x \in \mathbb{R}^{n}$ an input and $x' \in ...
user avatar
1 vote
0 answers
38 views

Why would one prefer the gradient of the sum rather than the sum of the gradients?

When gradients are aggregated over mini batches, I sometimes see formulations like this, e.g., in the "Deep Learning" book by Goodfellow et al. $$\mathbf{g} = \frac{1}{m} \nabla_{\mathbf{w}} ...
user avatar
  • 11
3 votes
1 answer
63 views

In general - is Stochastic Gradient Descent a "superior" algorithm compared to Gradient Descent? [closed]

On a very informal level, if we were to compare the (classical) Gradient Descent Algorithm to the Stochastic Gradient Descent Algorithm, the first thing that comes to mind is: Gradient Descent can be ...
user avatar
0 votes
1 answer
38 views

Unclear fact about difference between Gradient Descent to Stochastic Gradient Decent in wikipedia

From wikipedia page it mentioned: To economize on the computational cost at every iteration, stochastic gradient descent samples a subset of summand functions at every step. This is very effective in ...
user avatar
0 votes
0 answers
22 views

Does Stochastic Gradient Descent "Work" on (Some) Non-Convex Functions?

As we know, there has been a lot of work and research done to demonstrate that the Gradient Descent Algorithm can converge on (deterministic) convex, differentiable and Lipschitz Continuous functions :...
user avatar
0 votes
1 answer
27 views

Simple Polynomial Gradient Descent algorithm not working

I am trying to implement a simple 2nd order polynomial gradient descent algorithm in Java. It is not converging and becomes unstable. How do I fix it? ...
user avatar
0 votes
0 answers
24 views

How to calculate the gradient (or derivative) of y = f(x) of y w.r.t x where y represents the order statistics divided by median of x?

How to calculate the gradient (or derivative) of y = f(x) of y w.r.t x where y represents the order statistics divided by median of x? For instance x is ...
user avatar
  • 101
0 votes
0 answers
31 views

Limit of momentum update equation

I am self-studying on optimization algorithm on https://d2l.ai/chapter_optimization/momentum.html and couldn't get my head around some derivation: Instead of the standard gradient descent update ...
user avatar
2 votes
0 answers
54 views

Watkins' Q(λ) with function approximation: why is gradient not considered when updating eligibility traces for the exploitation phase?

I'm implementing the Watkins' Q(λ) algorithm with function approximation (in 2nd edition of Sutton & Barto). I am very confused about updating the eligibility traces because, at the beginning of ...
user avatar
1 vote
0 answers
46 views

GANs: Why does iterative gradient descent sometimes optimise $\min_G \max_D V(D,G)$ and sometimes $\max_D \min_G V(D,G)$?

For the following minimax equation for generative adversarial networks (GANs), $$\min_G \max_D V(D,G) = \mathbb{E}_{\boldsymbol{x}\sim p_{data}(\boldsymbol{x})}[\log D(\boldsymbol{x})] + \mathbb{E}_{\...
user avatar
5 votes
2 answers
381 views

Why is gradient descent used over the conjugate gradient method?

Based on some preliminary research, the conjugate gradient method is almost exactly the same as gradient descent, except the search direction must be orthogonal to the previous step. From what I've ...
user avatar
  • 1,324
1 vote
1 answer
37 views

Is it possible to use stochastic gradient descent at the beginning, then switch to batch gradient descent with only a few training examples?

Batch gradient descent is extremely slow for large datasets, but it can find the lowest possible value for the cost function. Stochastic gradient descent is relatively fast, but it kind of finds the ...
user avatar
  • 121
1 vote
0 answers
47 views

Why are optimization algorithms for deep learning so simple?

From my knowledge, the most used optimizer in practice is Adam, which in essence is just mini-batch gradient descent with momentum to combat getting stuck in saddle points and with some damping to ...
user avatar
1 vote
1 answer
365 views

What is uncentered variance and how it becomes equal to mean square in Adam?

I have been reading about Adam and AdamW (Here). The author mentioned that in "uncentered variance" we don't consider subtracting mean In this statement, the author is talking about ...
user avatar
  • 151
2 votes
0 answers
64 views

How to train neural networks with multiprocessing?

I am trying to figure out how multiprocessing works in neural networks. In the example I've seen, the database is split into $x$ parts (depending on how many workers you have) and each worker is ...
user avatar
0 votes
0 answers
42 views

In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?

Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one ...
user avatar
0 votes
0 answers
75 views

What is the effect of gradient clipping by norm on the performance of a model?

It is recommended to apply gradient clipping by normalization in case of exploding gradients. The following quote is taken from here answer One way to assure it is exploding gradients is if the loss ...
user avatar
  • 3,029
1 vote
1 answer
82 views

What is the difference between gradient decent in neural networks and temporal difference in reinforcement learning?

I am studying Q-learning in reinforcement learning. My question is about the Bellman equation. In Q-learning, the Bellman equation is often introduced as follows. \begin{align} Q_{new}(s,a) &= Q_{...
user avatar
1 vote
1 answer
38 views

How many iterations of the optimisation algorithm are performed on each mini-batch in mini-batch gradient descent?

I understand the idea of mini-batch gradient descent for neural networks in that we calculate the gradient of the loss function using one mini-batch at a time and use this gradient to adjust the ...
user avatar
0 votes
2 answers
47 views

In mini-batch gradient descent, do we pass each input in the batch individually or all inputs at the same time through the layer?

In the stochastic gradient descent algorithm, the weight update happens for every training sample. In the mini-batch gradient descent algorithm, the weight update happens for every batch of training ...
user avatar
  • 3,029
1 vote
1 answer
119 views

Different ways to calculate backpropagation derivatives, any difference?

I'm studying error backpropagation in neural networks. I am interested in why we use only one path on the computational graph to get the value of the derivative for a weight? I ask the question ...
user avatar
1 vote
1 answer
43 views

Do gradient-based algorithms deal with the flat regions with desired points?

I am studying a chapter named Numerical Computation of a deep learning book. Afaik, it does not deal with flat regions with desired points. For example, let us consider a function whose local/global ...
user avatar
  • 3,029
1 vote
0 answers
36 views

Is there any significance for higher order gradients in artificial intelligence?

Although I don't know in detail, I am aware of the following facts regarding the use of gradients in some domains of artificial intelligence, especially in minimizing the training of neural networks. ...
user avatar
  • 3,029
0 votes
2 answers
36 views

Reason for relaxing limit in derivative in this context?

Consider the following paragraph from NUMERICAL COMPUTATION of the deep learning book.. Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real numbers. The derivative of this function ...
user avatar
  • 3,029
0 votes
0 answers
12 views

What is meant by non-convergent limit cycles?

Limit cycle is a closed curve that is isolated i.e., no other closed curve near to it. You can read the following paragraph from here If there is (such) a closed curve, the nearby trajectories must ...
user avatar
  • 3,029
0 votes
0 answers
33 views

Is there any geometrical interpretation on overcoming gradient related problems by adjusting/changing loss function?

There are instances in literature where we need to change loss function in order to escape from gradient problems. Let $L_f$ be a loss function for a model I need to train on. Some times $L_f$ leads ...
user avatar
  • 3,029
1 vote
1 answer
73 views

Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?

While training a neural network, we can follow three methods: batch gradient descent, mini-batch gradient descent and stochastic gradient descent. For this question, assume that your dataset has $n$ ...
user avatar
  • 3,029
3 votes
1 answer
101 views

What are the necessary mathematical properties to be a loss function in gradient based optimizations?

Loss functions are used in training neural networks. I am interested in knowing the mathematical properties that are necessary for a loss function to participate in gradient descent optimization. I ...
user avatar
  • 3,029
3 votes
3 answers
881 views

In logistic regression, why is the binary cross-entropy loss function convex?

I am studying logistic regression for binary classification. The loss function used is cross-entropy. For a given input $x$, if our model outputs $\hat{y}$ instead of $y$, the loss is given by $$\text{...
user avatar
  • 3,029
0 votes
1 answer
177 views

Why is loss displayed as a parabola in mean squared error with gradient descent?

I'm looking at the loss function: mean squared error with gradient descent in machine learning. I'm building a single-neuron network (perceptron) that outputs a linear number. For example: Input * ...
user avatar
  • 167
1 vote
0 answers
43 views

Could the inputs of the mean squared-error loss function be transformed to allow larger learning rates?

In the context of a neural network $\hat{y} = f_\theta(\mathbf{x})$ with parameters $\theta$ that is trained to perform regression such that the prediction $\hat{\mathbf{y}} = [\hat{y}_1,\hat{y}_2,...,...
user avatar
  • 223
-1 votes
1 answer
43 views

How can the gradient of the weight be calculated in the viewpoint of matrix calculus?

Let $\sigma(x)$ be sigmoid function. Consider the case where $\text{out}=\sigma(\vec{x} \times W + \vec{b})$, and we want to compute $\frac{\partial{\text{out}}}{\partial{w} }.$ Set the dimension as ...
user avatar
5 votes
3 answers
253 views

Does gradient descent in deep learning assume a smooth fitness landscape?

I've come across the concept of fitness landscape before and, in my understanding, a smooth fitness landscape is one where the algorithm can converge on the global optimum through incremental ...
user avatar
  • 159
1 vote
0 answers
36 views

Optimizer that prevents parameters from oscillating

When we perform gradient descent, especially in an online setting where the training data is presented in a non-random order, a particular 1-dimensional parameter (such as an edge weight) may first ...
user avatar
  • 141
1 vote
0 answers
30 views

Do Gradient Descent and Natural Gradient solve the same problem?

I am troubled by natural gradient methods. If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do? I found on https://...
user avatar
  • 11
1 vote
1 answer
78 views

In gradient descent's update rule, why do we use $\sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}$ instead of $\frac{\delta C_0}{\delta w^{l}}$?

I am trying to code a two layered neural network simple NN as I have described here https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/ I am getting stuck on the last step of updating the ...
user avatar
2 votes
1 answer
288 views

How to deal with losses on different scales in multi-task learning?

Say I'm training a model for multiple tasks by trying to minimize sum of losses $L_1 + L_2$ via gradient descent. If these losses are on a different scale, the one whose range is greater will dominate ...
user avatar
2 votes
0 answers
31 views

What is the name of this algorithm that estimates the gradient with an average by sampling from a distribution?

Consider maximizing the function $R(w)$ with parameter $w$ using gradient ascent. However, we don't know the gradient $\nabla_wR(w)$ formula. Now suppose $w$ is sampled from a probability distribution ...
user avatar
  • 121
2 votes
0 answers
119 views

Bias gradient of layer before batch normalization always zero

From the original paper and this post we have that batch normalization backpropagation can be formulated as I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
user avatar
8 votes
2 answers
416 views

Is there an ideal range of learning rate which always gives a good result almost in all problems?

I once read somewhere that there is a range of learning rate within which learning is optimal in almost all the cases, but I can't find any literature about it. All I could get is the following graph ...
user avatar
2 votes
0 answers
57 views

Should I use batch gradient descent when I have a small sample size?

I have a dataset with an input size of 155x155, with the output being 155 x 1 with a 3-4 layer neural net being used for regression. With such a small sample size, should I use full batch gradient ...
user avatar
  • 29
4 votes
0 answers
93 views

Why are most commonly used activation functions continuous?

I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with ...
user avatar
2 votes
0 answers
28 views

Is it possible to ensure the convergence when training a RNN weight on its SVD decomposition?

I'm reading the following paper in which the author seems to do 2 things interesting: The hidden-to-hidden weight matrix of the RNN is SVD decomposed and train separately. Each orthogonal part of the ...
user avatar
1 vote
0 answers
31 views

How to derive compact convex set K and its diameter D to program Accelegrad algorithm in practice?

Given the original paper (https://arxiv.org/pdf/1809.02864.pdf), I would like to implement the Accelegrad algorithm for which I report the pseudocode of the paper: In the pseudocode, the authors ...
user avatar
  • 149
0 votes
1 answer
98 views

Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

If the derivative is supposed to give the rate of change of a function at that point, then why is the derivative of the softmax layer (a vector) the Jacobian matrix, which has a different shape than ...
user avatar
0 votes
1 answer
72 views

What is the derivative of a specific output with respect to a specific weight?

If I have a neural network, and say the 6th output node of the neural network is: $$x_6 = w_{16}y_1 + w_{26}y_2 + w_{36}y_3$$ What does that make the derivative of: $$\frac{\partial x_6}{\partial w_{...
user avatar
2 votes
2 answers
88 views

Is there anything that ensures that convolutional filters end up different from one another?

I found this question very interesting, and this is a follow up on it. Presumably, we'd want all the filters to converge towards some complementary set, where each filter fills as large a niche as ...
user avatar