Questions tagged [gradient-descent]
For questions surrounding gradient descent, a method for finding the optimum state of a parameterized function based on another function often called the loss or error function. It iteratively descends the loss surface to the minimum loss by adjusting parameters based on the product of the partial derivatives comprising the gradient and a learning rate.
190
questions
0
votes
0
answers
20
views
Problem about Xavier Initialization
I'm learning about Xavier initialization and reading paper "Understanding the difficulty of training deep feedforward neural networks". https://www.researchgate.net/publication/...
- 1
-1
votes
1
answer
26
views
Why do we need to find derivative for activation function?
I am new in AI/ML. My question is pretty straight. I am learning about Gradient Descent, which is, in my understanding, a way to minimize "errors" and make our model more robust. I ...
0
votes
0
answers
49
views
Why is cross entropy loss averaged and not used directly as a sum during model training?
Why is the cross-entropy loss for all training examples (or the training examples in a batch) averaged over the size of the training set (or batch size)?
Why is it not just summed up and used?
- 1
0
votes
0
answers
31
views
Is Momentum a form of inexact line search?
Is Momentum a form of inexact line search? I just want to know whether Momentum is considered a subset of inexact line search.
By, inexact I mean that the step size is not the perfect one that ...
0
votes
1
answer
114
views
Why to use gradient accumulation?
I know that gradient accumulation is (1) a way to reduce memory usage while still enabling the machine to fit a large dataset (2) reducing the noise of the gradient compared to SGD, and thus smoothing ...
- 101
0
votes
1
answer
38
views
Single Layer Perceptron Backpropagation: How to compute affect of the net value on the output?
Assuming a single perceptron (see figure), I have found two versions of how to use backpropagation to update the weights. The perceptron is split in two, so we see the weighted sum on the left (the ...
- 1
2
votes
2
answers
64
views
How are gradients of individual layers computed?
I have been reading some papers recently (example: https://arxiv.org/pdf/2012.00363.pdf) which seem to be training individual layers of, say, a transformer, holding the rest of the model frozen/...
- 21
0
votes
1
answer
21
views
What are your "current parameters" in Minibatch Stochastic Gradient Descent?
I was reading a book on Deep Learning when I came across a line, more like a few words that didn't make apparent sense.
Thus, we will often settle for sampling a random minibatch of examples every ...
- 101
1
vote
2
answers
62
views
Can I minimize a mysterious function by running a gradient descent on her neural net approximations?
So I have this function let call her $F:[0,1]^n \rightarrow \mathbb{R}$ and say $10 \le n \le 100$. I want to find some $x_0 \in [0,1]^n$ such that $F(x_0)$ is as small as possible. I don't think ...
- 111
0
votes
1
answer
57
views
During batch normalization is the mini-batch gone through twice, one to calculate the mean and variance and then to normalize them?
I am asking this question because while designing my own model, I had repeated gradient explosion issues, so I wanted to try batch normalization. I really want to understand the details and math ...
0
votes
1
answer
77
views
Numerical problems with gradient descent
I'm trying to implement a simple neural network for classification (multi-class) as an exercise (written in C). During gradient descent, the weights and biases quickly get out of control and the ...
- 233
2
votes
0
answers
41
views
Can objective function and gradient be unlimited in reinforcement learning?
I'm looking at an example where they define a policy $\pi_\theta(a_t|s_t)\sim \mathcal{N}(ks_t, \sigma)$, where $a_t$ and $s_t$ are action and state, while $\theta=(k,\sigma)$ are the parameters of ...
- 41
1
vote
1
answer
120
views
Why do we use gradient descent to minimize the loss function?
The purpose of training neural networks is to minimize a loss function, in this process we usually use gradient descent method.
But in Calculus, if we want to find the global minimum of a ...
- 111
0
votes
1
answer
99
views
What exactly is the AI explainability problem?
I am pretty new to AI and have recently been paying attention to AI explainability and the fact that it remains a hurdle within the path of commercializing certain AI systems in health for instance. I ...
- 1
1
vote
1
answer
305
views
How does MAML inner loop optimization works?
I started to learn meta-learning, reading the MAML paper https://arxiv.org/pdf/1703.03400.pdf
In the inner loop, I am calculating adapted parameters for each task, I will be doing multiple steps of ...
- 43
1
vote
2
answers
59
views
At which point, does the momentum based GD helps really in this figure?
Classical gradient descent algorithms sometimes overshoot and escape minima as they depend on the gradient only. You can see such a problem during the update from point 6.
In classical GD algorithm, ...
- 3,481
0
votes
0
answers
27
views
Resolving Derivation Discrepancies for Differentiating through Optimization Paths
I'm reading the paper "Optimizing Millions of Hyperparameters by Implicit Differentiation". The key contribution of the paper is to show that you can replace optimizing through the ...
- 1
0
votes
1
answer
175
views
How to explain near zero gradients on first epochs?
As I understand the gradient should reflect how near the weights are to the optimal values. In this way i will expect that on the first epochs the gradients far from zero or at least not mostly zero ...
0
votes
0
answers
21
views
How to interpret integrated gradients in an NLP toxic text classification use-case?
I am trying to understand how integrated gradients work in the NLP case.
Let $F: \mathbb{R}^{n} \rightarrow[0,1]$ a function representing a neural network, $x \in \mathbb{R}^{n}$ an input and $x' \in ...
1
vote
0
answers
168
views
Why would one prefer the gradient of the sum rather than the sum of the gradients?
When gradients are aggregated over mini batches, I sometimes see formulations like this, e.g., in the "Deep Learning" book by Goodfellow et al.
$$\mathbf{g} = \frac{1}{m} \nabla_{\mathbf{w}} ...
- 11
3
votes
1
answer
120
views
In general - is Stochastic Gradient Descent a "superior" algorithm compared to Gradient Descent? [closed]
On a very informal level, if we were to compare the (classical) Gradient Descent Algorithm to the Stochastic Gradient Descent Algorithm, the first thing that comes to mind is:
Gradient Descent can be ...
- 249
0
votes
1
answer
76
views
Unclear fact about difference between Gradient Descent to Stochastic Gradient Decent in wikipedia
From wikipedia page it mentioned:
To economize on the computational cost at every iteration, stochastic
gradient descent samples a subset of summand functions at every step.
This is very effective in ...
- 121
0
votes
1
answer
33
views
Simple Polynomial Gradient Descent algorithm not working
I am trying to implement a simple 2nd order polynomial gradient descent algorithm in Java. It is not converging and becomes unstable. How do I fix it?
...
- 111
2
votes
0
answers
110
views
Watkins' Q(λ) with function approximation: why is gradient not considered when updating eligibility traces for the exploitation phase?
I'm implementing the Watkins' Q(λ) algorithm with function approximation (in 2nd edition of Sutton & Barto).
I am very confused about updating the eligibility traces because, at the beginning of ...
1
vote
0
answers
51
views
GANs: Why does iterative gradient descent sometimes optimise $\min_G \max_D V(D,G)$ and sometimes $\max_D \min_G V(D,G)$?
For the following minimax equation for generative adversarial networks (GANs),
$$\min_G \max_D V(D,G) = \mathbb{E}_{\boldsymbol{x}\sim p_{data}(\boldsymbol{x})}[\log D(\boldsymbol{x})] + \mathbb{E}_{\...
- 23
5
votes
2
answers
2k
views
Why is gradient descent used over the conjugate gradient method?
Based on some preliminary research, the conjugate gradient method is almost exactly the same as gradient descent, except the search direction must be orthogonal to the previous step.
From what I've ...
- 1,316
1
vote
1
answer
75
views
Is it possible to use stochastic gradient descent at the beginning, then switch to batch gradient descent with only a few training examples?
Batch gradient descent is extremely slow for large datasets, but it can find the lowest possible value for the cost function. Stochastic gradient descent is relatively fast, but it kind of finds the ...
- 121
1
vote
0
answers
75
views
Why are optimization algorithms for deep learning so simple?
From my knowledge, the most used optimizer in practice is Adam, which in essence is just mini-batch gradient descent with momentum to combat getting stuck in saddle points and with some damping to ...
- 133
1
vote
1
answer
861
views
What is uncentered variance and how it becomes equal to mean square in Adam?
I have been reading about Adam and AdamW (Here). The author mentioned that in "uncentered variance" we don't consider subtracting mean
In this statement, the author is talking about ...
- 151
2
votes
0
answers
194
views
How to train neural networks with multiprocessing?
I am trying to figure out how multiprocessing works in neural networks.
In the example I've seen, the database is split into $x$ parts (depending on how many workers you have) and each worker is ...
- 121
0
votes
1
answer
331
views
In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?
Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one ...
1
vote
0
answers
246
views
What is the effect of gradient clipping by norm on the performance of a model?
It is recommended to apply gradient clipping by normalization in case of exploding gradients. The following quote is taken from here answer
One way to assure it is exploding gradients is if the loss ...
- 3,481
1
vote
1
answer
158
views
What is the difference between gradient decent in neural networks and temporal difference in reinforcement learning?
I am studying Q-learning in reinforcement learning. My question is about the Bellman equation.
In Q-learning, the Bellman equation is often introduced as follows.
\begin{align}
Q_{new}(s,a)
&= Q_{...
- 111
1
vote
1
answer
202
views
How many iterations of the optimisation algorithm are performed on each mini-batch in mini-batch gradient descent?
I understand the idea of mini-batch gradient descent for neural networks in that we calculate the gradient of the loss function using one mini-batch at a time and use this gradient to adjust the ...
- 13
1
vote
2
answers
266
views
In mini-batch gradient descent, do we pass each input in the batch individually or all inputs at the same time through the layer?
In the stochastic gradient descent algorithm, the weight update happens for every training sample.
In the mini-batch gradient descent algorithm, the weight update happens for every batch of training ...
- 3,481
1
vote
1
answer
395
views
Different ways to calculate backpropagation derivatives, any difference?
I'm studying error backpropagation in neural networks. I am interested in why we use only one path on the computational graph to get the value of the derivative for a weight? I ask the question ...
- 13
2
votes
1
answer
169
views
Do gradient-based algorithms deal with the flat regions with desired points?
I am studying a chapter named Numerical Computation of a deep learning book. Afaik, it does not deal with flat regions with desired points.
For example, let us consider a function whose local/global ...
- 3,481
1
vote
1
answer
58
views
Is there any significance for higher order gradients in artificial intelligence?
Although I don't know in detail, I am aware of the following facts regarding the use of gradients in some domains of artificial intelligence, especially in minimizing the training of neural networks.
...
- 3,481
0
votes
2
answers
37
views
Reason for relaxing limit in derivative in this context?
Consider the following paragraph from NUMERICAL COMPUTATION of the deep learning book..
Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real
numbers. The derivative of this function ...
- 3,481
0
votes
0
answers
35
views
Is there any geometrical interpretation on overcoming gradient related problems by adjusting/changing loss function?
There are instances in literature where we need to change loss function in order to escape from gradient problems.
Let $L_f$ be a loss function for a model I need to train on. Some times $L_f$ leads ...
- 3,481
2
votes
1
answer
191
views
Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?
While training a neural network, we can follow three methods: batch gradient descent, mini-batch gradient descent and stochastic gradient descent.
For this question, assume that your dataset has $n$ ...
- 3,481
4
votes
1
answer
475
views
What are the necessary mathematical properties to be a loss function in gradient based optimizations?
Loss functions are used in training neural networks.
I am interested in knowing the mathematical properties that are necessary for a loss function to participate in gradient descent optimization.
I ...
- 3,481
3
votes
3
answers
3k
views
In logistic regression, why is the binary cross-entropy loss function convex?
I am studying logistic regression for binary classification.
The loss function used is cross-entropy. For a given input $x$, if our model outputs $\hat{y}$ instead of $y$, the loss is given by
$$\text{...
- 3,481
0
votes
1
answer
344
views
Why is loss displayed as a parabola in mean squared error with gradient descent?
I'm looking at the loss function: mean squared error with gradient descent in machine learning. I'm building a single-neuron network (perceptron) that outputs a linear number. For example:
Input * ...
- 167
1
vote
0
answers
68
views
Could the inputs of the mean squared-error loss function be transformed to allow larger learning rates?
In the context of a neural network $\hat{y} = f_\theta(\mathbf{x})$ with parameters $\theta$ that is trained to perform regression such that the prediction $\hat{\mathbf{y}} = [\hat{y}_1,\hat{y}_2,...,...
- 223
-1
votes
1
answer
164
views
How can the gradient of the weight be calculated in the viewpoint of matrix calculus?
Let $\sigma(x)$ be sigmoid function. Consider the case where $\text{out}=\sigma(\vec{x} \times W + \vec{b})$, and we want to compute $\frac{\partial{\text{out}}}{\partial{w}
}.$
Set the dimension as ...
- 99
5
votes
3
answers
318
views
Does gradient descent in deep learning assume a smooth fitness landscape?
I've come across the concept of fitness landscape before and, in my understanding, a smooth fitness landscape is one where the algorithm can converge on the global optimum through incremental ...
- 159
1
vote
0
answers
42
views
Optimizer that prevents parameters from oscillating
When we perform gradient descent, especially in an online setting where the training data is presented in a non-random order, a particular 1-dimensional parameter (such as an edge weight) may first ...
- 141
1
vote
0
answers
67
views
Do Gradient Descent and Natural Gradient solve the same problem?
I am troubled by natural gradient methods.
If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do?
I found on https://...
- 11
1
vote
1
answer
143
views
In gradient descent's update rule, why do we use $\sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}$ instead of $\frac{\delta C_0}{\delta w^{l}}$?
I am trying to code a two layered neural network simple NN as I have described here https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/
I am getting stuck on the last step of updating the ...
- 121