For questions surrounding gradient descent, a method for finding the optimum state of a parameterized function based on another function often called the loss or error function. It iteratively descends the loss surface to the minimum loss by adjusting parameters based on the product of the partial derivatives comprising the gradient and a learning rate.

201 questions
Filter by
Sorted by
Tagged with
6 views

### NaN gradients while Training (but loss isn't NaN neither the computational graph is disconnected)

The reason I am asking this here is that I haven't found a bug in my code and maybe there isn't a bug at all (or maybe there is). I just want to validate the idea that I am trying to implement. Here ...
1 vote
38 views

### A Feedforward Neural Network (FNN) implemented with RMSProp optimization is exhibiting a tendency to overclassify instances into one particular class

hope you're alright. I'm coding an FNN in Rust using the nalgebra crate. I coded the backpropagation based on this article from Brilliant (the link directly highlights the formulas' section I). The ...
27 views

### What are w0 and w1 respectively after training with the following two examples of (x, y) in the given order?

The answer is supposed to be w0 = 2.8229, w1 = 2.4686. I'm not sure how that is the case. Can you please also show how you arrived at the solution?
35 views

I have read some resources about AI, and they all speak about the gradient. Is there any book focused on this? maybe with tons of images / diagrams? Cheers
20 views

### Independent parameter update in backpropagation

When we calculate the gradient wrt to each paramters, we consider the other parameters remain constant, but the moment their is a change in any of the other parameters, shouldn't all the other changes ...
28 views

### What are the differences between loss surfaces that "derive"from different observations?

If I understand right that each observation whithin a dataset, creates a different loss surface where we want to find the global minimum. How different those surfaces one from another? Would it be ...
16 views

### Relation between the number of parameters and the features in Gradient descent algorithm

My book describes this as an equation for minimizing the $\theta$ value, but I have a few questions regarding the intuition behind this equation: The book describes $j$ as the number of features. If ...
1 vote
31 views

### Can gradient descent cause loss to increase in some situations?

Is a gradient descent step always supposed to decrease loss? I can think of a situation where it would seem that gradient descent would increase loss but maybe it I am misunderstanding a part of ...
1 vote
39 views

### Is there a resource that offers a detailed overview of the gradient flow?

Understanding the concept of "Gradient Flow" can be quite difficult as there is a lack of widely recognized and clearly defined resources that provide a comprehensive explanation. Although ...
67 views

### Is there a recommended resource that can provide a detailed overview of the gradient norm?

When it comes to the concept of "Gradient Norm," it can be challenging to find a widely recognized and clearly defined resource that offers a comprehensive explanation. While many search ...
747 views

### Why use ReLU over Leaky ReLU?

From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive. With just this info to go off of,...
208 views

### Combine multiple losses with gradient descent

I am optimizing a neural network with Adam using 3 different losses. Their scale is very different, and the current method is to either sum the losses and clip the gradient or to manually weight them ...
120 views

### What is the justification for this approach of clipping elementwise?

I'm new to the field of AI (though I have a background in mathematics). As I was going through some documents, I read that there is a form of gradient clipping where the elements of the gradient that ...
146 views

### How does gradient descent work with relu if weights are negative?

How does gradient descent work with relu, imagine the weights are quite negative and so our "prediction" is 0, then not much is learned. Is there a risk that training gets stuck when weights ...
66 views

### Why is cross entropy loss averaged and not used directly as a sum during model training?

Why is the cross-entropy loss for all training examples (or the training examples in a batch) averaged over the size of the training set (or batch size)? Why is it not just summed up and used?
43 views

### Is Momentum a form of inexact line search?

Is Momentum a form of inexact line search? I just want to know whether Momentum is considered a subset of inexact line search. By, inexact I mean that the step size is not the perfect one that ...
1k views

### Why to use gradient accumulation?

I know that gradient accumulation is (1) a way to reduce memory usage while still enabling the machine to fit a large dataset (2) reducing the noise of the gradient compared to SGD, and thus smoothing ...
258 views

### Single Layer Perceptron Backpropagation: How to compute affect of the net value on the output?

Assuming a single perceptron (see figure), I have found two versions of how to use backpropagation to update the weights. The perceptron is split in two, so we see the weighted sum on the left (the ...
117 views

### How are gradients of individual layers computed?

I have been reading some papers recently (example: https://arxiv.org/pdf/2012.00363.pdf) which seem to be training individual layers of, say, a transformer, holding the rest of the model frozen/...
26 views

I was reading a book on Deep Learning when I came across a line, more like a few words that didn't make apparent sense. Thus, we will often settle for sampling a random minibatch of examples every ...
1 vote
66 views

### Can I minimize a mysterious function by running a gradient descent on her neural net approximations?

So I have this function let call her $F:[0,1]^n \rightarrow \mathbb{R}$ and say $10 \le n \le 100$. I want to find some $x_0 \in [0,1]^n$ such that $F(x_0)$ is as small as possible. I don't think ...
84 views

### During batch normalization is the mini-batch gone through twice, one to calculate the mean and variance and then to normalize them?

I am asking this question because while designing my own model, I had repeated gradient explosion issues, so I wanted to try batch normalization. I really want to understand the details and math ...
101 views

### Numerical problems with gradient descent

I'm trying to implement a simple neural network for classification (multi-class) as an exercise (written in C). During gradient descent, the weights and biases quickly get out of control and the ...
50 views

### Can objective function and gradient be unlimited in reinforcement learning?

I'm looking at an example where they define a policy $\pi_\theta(a_t|s_t)\sim \mathcal{N}(ks_t, \sigma)$, where $a_t$ and $s_t$ are action and state, while $\theta=(k,\sigma)$ are the parameters of ...
1 vote
205 views

### Why do we use gradient descent to minimize the loss function?

The purpose of training neural networks is to minimize a loss function, in this process we usually use gradient descent method. But in Calculus, if we want to find the global minimum of a ...
141 views

### What exactly is the AI explainability problem?

I am pretty new to AI and have recently been paying attention to AI explainability and the fact that it remains a hurdle within the path of commercializing certain AI systems in health for instance. I ...
1 vote
527 views

### How does MAML inner loop optimization works?

I started to learn meta-learning, reading the MAML paper https://arxiv.org/pdf/1703.03400.pdf In the inner loop, I am calculating adapted parameters for each task, I will be doing multiple steps of ...
1 vote
75 views

### At which point, does the momentum based GD helps really in this figure?

Classical gradient descent algorithms sometimes overshoot and escape minima as they depend on the gradient only. You can see such a problem during the update from point 6. In classical GD algorithm, ...
27 views

### Resolving Derivation Discrepancies for Differentiating through Optimization Paths

I'm reading the paper "Optimizing Millions of Hyperparameters by Implicit Differentiation". The key contribution of the paper is to show that you can replace optimizing through the ...
344 views

### How to explain near zero gradients on first epochs?

As I understand the gradient should reflect how near the weights are to the optimal values. In this way i will expect that on the first epochs the gradients far from zero or at least not mostly zero ...
1 vote
346 views

When gradients are aggregated over mini batches, I sometimes see formulations like this, e.g., in the "Deep Learning" book by Goodfellow et al. $$\mathbf{g} = \frac{1}{m} \nabla_{\mathbf{w}} ... 3 votes 1 answer 184 views ### In general - is Stochastic Gradient Descent a "superior" algorithm compared to Gradient Descent? [closed] On a very informal level, if we were to compare the (classical) Gradient Descent Algorithm to the Stochastic Gradient Descent Algorithm, the first thing that comes to mind is: Gradient Descent can be ... 0 votes 1 answer 97 views ### Unclear fact about difference between Gradient Descent to Stochastic Gradient Decent in wikipedia From wikipedia page it mentioned: To economize on the computational cost at every iteration, stochastic gradient descent samples a subset of summand functions at every step. This is very effective in ... 0 votes 1 answer 44 views ### Simple Polynomial Gradient Descent algorithm not working I am trying to implement a simple 2nd order polynomial gradient descent algorithm in Java. It is not converging and becomes unstable. How do I fix it? ... 2 votes 0 answers 141 views ### Watkins' Q(λ) with function approximation: why is gradient not considered when updating eligibility traces for the exploitation phase? I'm implementing the Watkins' Q(λ) algorithm with function approximation (in 2nd edition of Sutton & Barto). I am very confused about updating the eligibility traces because, at the beginning of ... 1 vote 0 answers 55 views ### GANs: Why does iterative gradient descent sometimes optimise \min_G \max_D V(D,G) and sometimes \max_D \min_G V(D,G)? For the following minimax equation for generative adversarial networks (GANs),$$\min_G \max_D V(D,G) = \mathbb{E}_{\boldsymbol{x}\sim p_{data}(\boldsymbol{x})}[\log D(\boldsymbol{x})] + \mathbb{E}_{\...
4k views

Based on some preliminary research, the conjugate gradient method is almost exactly the same as gradient descent, except the search direction must be orthogonal to the previous step. From what I've ...
1 vote
94 views

### Is it possible to use stochastic gradient descent at the beginning, then switch to batch gradient descent with only a few training examples?

Batch gradient descent is extremely slow for large datasets, but it can find the lowest possible value for the cost function. Stochastic gradient descent is relatively fast, but it kind of finds the ...
1 vote
81 views

### Why are optimization algorithms for deep learning so simple?

From my knowledge, the most used optimizer in practice is Adam, which in essence is just mini-batch gradient descent with momentum to combat getting stuck in saddle points and with some damping to ...
1 vote
1k views

### What is uncentered variance and how it becomes equal to mean square in Adam?

I have been reading about Adam and AdamW (Here). The author mentioned that in "uncentered variance" we don't consider subtracting mean In this statement, the author is talking about ...
315 views

### How to train neural networks with multiprocessing?

I am trying to figure out how multiprocessing works in neural networks. In the example I've seen, the database is split into $x$ parts (depending on how many workers you have) and each worker is ...
468 views

### In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?

Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one ...
1 vote
475 views

### What is the effect of gradient clipping by norm on the performance of a model?

It is recommended to apply gradient clipping by normalization in case of exploding gradients. The following quote is taken from here answer One way to assure it is exploding gradients is if the loss ...
1 vote
222 views

### What is the difference between gradient decent in neural networks and temporal difference in reinforcement learning?

I am studying Q-learning in reinforcement learning. My question is about the Bellman equation. In Q-learning, the Bellman equation is often introduced as follows. \begin{align} Q_{new}(s,a) &= Q_{...
1 vote
287 views

### How many iterations of the optimisation algorithm are performed on each mini-batch in mini-batch gradient descent?

I understand the idea of mini-batch gradient descent for neural networks in that we calculate the gradient of the loss function using one mini-batch at a time and use this gradient to adjust the ...
1 vote
439 views

### In mini-batch gradient descent, do we pass each input in the batch individually or all inputs at the same time through the layer?

In the stochastic gradient descent algorithm, the weight update happens for every training sample. In the mini-batch gradient descent algorithm, the weight update happens for every batch of training ...
1 vote
669 views

### Different ways to calculate backpropagation derivatives, any difference?

I'm studying error backpropagation in neural networks. I am interested in why we use only one path on the computational graph to get the value of the derivative for a weight? I ask the question ...
308 views

### Do gradient-based algorithms deal with the flat regions with desired points?

I am studying a chapter named Numerical Computation of a deep learning book. Afaik, it does not deal with flat regions with desired points. For example, let us consider a function whose local/global ...
1 vote
Consider the following paragraph from NUMERICAL COMPUTATION of the deep learning book.. Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real numbers. The derivative of this function ...