Skip to main content

Questions tagged [gradient-descent]

For questions surrounding gradient descent, a method for finding the optimum state of a parameterized function based on another function often called the loss or error function. It iteratively descends the loss surface to the minimum loss by adjusting parameters based on the product of the partial derivatives comprising the gradient and a learning rate.

Filter by
Sorted by
Tagged with
0 votes
0 answers
36 views

How is this z-loss implementation in t5x related to this paper's loss X?

I was looking into the loss function in t5x here and see there is a z-loss added to the typical log loss definition. The only paper I could surface on this was https://arxiv.org/abs/1604.08859, but I ...
Jacob B's user avatar
  • 247
0 votes
0 answers
13 views

Nesterov Accelerated Gradient is Slower than Momentum [closed]

I'm working on really simple FFNN on Fashion MNIST to test out different optimizers by implementing them from scratch in NumPy. I'm finding that Nesterov Accelerated Gradient is slower than regular ...
vxnuaj's user avatar
  • 115
2 votes
1 answer
29 views

Different Definitions of Momentum -- which one should I work with?

I'm seeing different manners to define momentum, I'm not sure if there is significant difference or not. From my thinking, they seem to do a similar thing mathematically and in practice but I'm ...
vxnuaj's user avatar
  • 115
0 votes
0 answers
33 views

Learning Rate greater than ~0.00005 significantly hinders model performance and increases loss

I have been trying to train a model with 0.001 learning rate. I tried regression techniques, early stopping and lr manipulations within epochs. But nothing felt right even though after numerous tries ...
Yigithan Sever's user avatar
0 votes
0 answers
40 views

custom neural network implementation is giving 10% accuracy on mnist dataset

I've created a toy neural network in python for learning purposes and decided to test it on mnist dataset, I didn't expect great results but the result that I got - 10% is as good as a guess. For many ...
bearthum's user avatar
0 votes
0 answers
25 views

Not Averaging Gamma and Beta Gradients in BatchNormalization leads me to higher accuracy

I'm implementing batchnorm from scratch in pure NumPy. I noticed something interesting. While I'm calculating the gradients of gamma (dg) and beta (db), ignoring the summation / averaging of the ...
vxnuaj's user avatar
  • 115
0 votes
0 answers
17 views

Covering missing derivations in Bengio RNN exploding gradient/output paper

While I have no doubts about the soundness of its conclusions given that it is very well-known, I have been looking through the classic paper on RNNs which is used to substantiate the claims that ...
Chris's user avatar
  • 101
0 votes
0 answers
11 views

collaborative filtering using linear regression

Currently doing andrew ng's unsupervised learning specialization, I came across this algorithm for collaborative filtering: here the Xi refers to feature vector of objects(ex: action in movies, ...
SRAVAN KOTTA's user avatar
0 votes
0 answers
22 views

Is there any purpose of altering neural network architecture if validation loss does not decrease but training loss does?

I am training a transformer based neural network and the validation loss is not decreasing, but the training loss does decrease. I am wondering if it's possible to debug or change the architecture ...
JobHunter69's user avatar
1 vote
1 answer
35 views

Why same learning rate for slope and intercept not working in Linear regression?

I'm a new student in AI, currently learning linear regression. I used the california housing dataset for doing my experiments. My goal is to predict the 'population' column based on the 'total_rooms' ...
Jahid Chowdhury Choton's user avatar
1 vote
1 answer
73 views

Is there a theoretical way to determine the best learning rate for gradient descent if the function is a simple known polynomial?

I was playing around gradient descent topic. Wrote a function that calculates a gradient descent of a degree-2 polynomial. While trying out what is the best "step size multiplyer" (a.k.a. &...
Ababababa's user avatar
  • 113
2 votes
1 answer
150 views

REINFORCE with Baseline update rule

I was looking at the algorithm for REINFORCE with baseline from the Book 'Introduction to Reinforcement Learning' from Sutton: I do not quite understand the update rule for $w$: $w = w + \alpha \...
kklaw's user avatar
  • 185
3 votes
1 answer
265 views

What do you mean by "updating based on a training example/batch" in Gradient Descent?

My understanding is this: When doing Stochastic Gradient Descent over a neural network, in every epoch, we run $n$ iterations (where the dataset has $n$ training examples) and in every iteration, we ...
insipidintegrator's user avatar
2 votes
1 answer
63 views

Can you explain the Hinton's comment "Rprop is equivalent to using the gradient, but also dividing by the size of the gradient"?

Been reviewing some old foundational material and ran into this comment by Hinton on Rprop in his old Coursera class: Rprop is equivalent to using the gradient, but also dividing by the size of the ...
eof's user avatar
  • 121
1 vote
2 answers
43 views

Why use learning rate schedules if weight updates automatically decrease when approaching local optimal?

Andrew Ng said in his slide that: However, there are numerous types of 'learning rate schedules' in TensorFlow that change the learning rate profile as training progresses. If it's true that these ...
Wong's user avatar
  • 11
0 votes
0 answers
32 views

NaN gradients while Training (but loss isn't NaN neither the computational graph is disconnected)

The reason I am asking this here is that I haven't found a bug in my code and maybe there isn't a bug at all (or maybe there is). I just want to validate the idea that I am trying to implement. Here ...
Ryukendo Dey's user avatar
1 vote
1 answer
56 views

A Feedforward Neural Network (FNN) implemented with RMSProp optimization is exhibiting a tendency to overclassify instances into one particular class

I'm coding an FNN in Rust using the nalgebra crate. I coded the backpropagation based on this article from Brilliant (the link directly highlights the formulas' section I). The issue My network tends ...
Evry's user avatar
  • 13
0 votes
0 answers
36 views

What are w0 and w1 respectively after training with the following two examples of (x, y) in the given order?

The answer is supposed to be w0 = 2.8229, w1 = 2.4686. I'm not sure how that is the case. Can you please also show how you arrived at the solution?
LuminousStar's user avatar
0 votes
1 answer
56 views

Gradient: any resource on how to understand everything about it?

I have read some resources about AI, and they all speak about the gradient. Is there any book focused on this? maybe with tons of images / diagrams? Cheers
zerunio's user avatar
0 votes
0 answers
32 views

Independent parameter update in backpropagation

When we calculate the gradient wrt to each paramters, we consider the other parameters remain constant, but the moment their is a change in any of the other parameters, shouldn't all the other changes ...
In progress...'s user avatar
0 votes
1 answer
32 views

What are the differences between loss surfaces that "derive"from different observations?

If I understand right that each observation whithin a dataset, creates a different loss surface where we want to find the global minimum. How different those surfaces one from another? Would it be ...
Igor's user avatar
  • 183
0 votes
0 answers
38 views

Relation between the number of parameters and the features in Gradient descent algorithm

My book describes this as an equation for minimizing the $\theta$ value, but I have a few questions regarding the intuition behind this equation: The book describes $j$ as the number of features. If ...
someman112's user avatar
1 vote
0 answers
43 views

Can gradient descent cause loss to increase in some situations?

Is a gradient descent step always supposed to decrease loss? I can think of a situation where it would seem that gradient descent would increase loss but maybe it I am misunderstanding a part of ...
Mike Levi's user avatar
2 votes
2 answers
66 views

Is there a resource that offers a detailed overview of the gradient flow?

Understanding the concept of "Gradient Flow" can be quite difficult as there is a lack of widely recognized and clearly defined resources that provide a comprehensive explanation. Although ...
v1998199904's user avatar
0 votes
1 answer
123 views

Is there a recommended resource that can provide a detailed overview of the gradient norm?

When it comes to the concept of "Gradient Norm," it can be challenging to find a widely recognized and clearly defined resource that offers a comprehensive explanation. While many search ...
StudentV's user avatar
11 votes
1 answer
5k views

Why use ReLU over Leaky ReLU?

From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive. With just this info to go off of,...
John Brown's user avatar
4 votes
1 answer
631 views

What is the best way to combine or weight multiple losses with gradient descent?

I am optimizing a neural network with Adam using 3 different losses. Their scale is very different, and the current method is to either sum the losses and clip the gradient or to manually weight them ...
Simon's user avatar
  • 153
2 votes
1 answer
329 views

What is the justification for this approach of clipping elementwise?

I'm new to the field of AI (though I have a background in mathematics). As I was going through some documents, I read that there is a form of gradient clipping where the elements of the gradient that ...
Ukn0wn's user avatar
  • 21
2 votes
1 answer
355 views

How does gradient descent work with relu if weights are negative?

How does gradient descent work with relu, imagine the weights are quite negative and so our "prediction" is 0, then not much is learned. Is there a risk that training gets stuck when weights ...
Dirk N's user avatar
  • 158
1 vote
1 answer
2k views

Why to use gradient accumulation?

I know that gradient accumulation is (1) a way to reduce memory usage while still enabling the machine to fit a large dataset (2) reducing the noise of the gradient compared to SGD, and thus smoothing ...
Cyrus's user avatar
  • 111
0 votes
1 answer
401 views

Single Layer Perceptron Backpropagation: How to compute affect of the net value on the output?

Assuming a single perceptron (see figure), I have found two versions of how to use backpropagation to update the weights. The perceptron is split in two, so we see the weighted sum on the left (the ...
HTH's user avatar
  • 1
2 votes
2 answers
204 views

How are gradients of individual layers computed?

I have been reading some papers recently (example: https://arxiv.org/pdf/2012.00363.pdf) which seem to be training individual layers of, say, a transformer, holding the rest of the model frozen/...
nlp4892's user avatar
  • 21
0 votes
1 answer
28 views

What are your "current parameters" in Minibatch Stochastic Gradient Descent?

I was reading a book on Deep Learning when I came across a line, more like a few words that didn't make apparent sense. Thus, we will often settle for sampling a random minibatch of examples every ...
HarshDarji's user avatar
1 vote
2 answers
68 views

Can I minimize a mysterious function by running a gradient descent on her neural net approximations?

So I have this function let call her $F:[0,1]^n \rightarrow \mathbb{R}$ and say $10 \le n \le 100$. I want to find some $x_0 \in [0,1]^n$ such that $F(x_0)$ is as small as possible. I don't think ...
Vladimir Zolotov's user avatar
0 votes
1 answer
96 views

During batch normalization is the mini-batch gone through twice, one to calculate the mean and variance and then to normalize them?

I am asking this question because while designing my own model, I had repeated gradient explosion issues, so I wanted to try batch normalization. I really want to understand the details and math ...
liyu zerihun's user avatar
0 votes
1 answer
139 views

Numerical problems with gradient descent

I'm trying to implement a simple neural network for classification (multi-class) as an exercise (written in C). During gradient descent, the weights and biases quickly get out of control and the ...
martinkunev's user avatar
2 votes
0 answers
50 views

Can objective function and gradient be unlimited in reinforcement learning?

I'm looking at an example where they define a policy $\pi_\theta(a_t|s_t)\sim \mathcal{N}(ks_t, \sigma)$, where $a_t$ and $s_t$ are action and state, while $\theta=(k,\sigma)$ are the parameters of ...
pippo's user avatar
  • 41
1 vote
1 answer
334 views

Why do we use gradient descent to minimize the loss function?

The purpose of training neural networks is to minimize a loss function, in this process we usually use gradient descent method. But in Calculus, if we want to find the global minimum of a ...
Proton's user avatar
  • 111
0 votes
1 answer
216 views

What exactly is the AI explainability problem?

I am pretty new to AI and have recently been paying attention to AI explainability and the fact that it remains a hurdle within the path of commercializing certain AI systems in health for instance. I ...
rp2001's user avatar
  • 1
1 vote
1 answer
666 views

How does MAML inner loop optimization works?

I started to learn meta-learning, reading the MAML paper https://arxiv.org/pdf/1703.03400.pdf In the inner loop, I am calculating adapted parameters for each task, I will be doing multiple steps of ...
Grumpy C's user avatar
1 vote
3 answers
98 views

At which point, does the momentum based GD helps really in this figure?

Classical gradient descent algorithms sometimes overshoot and escape minima as they depend on the gradient only. You can see such a problem during the update from point 6. In classical GD algorithm, ...
hanugm's user avatar
  • 3,930
0 votes
0 answers
28 views

Resolving Derivation Discrepancies for Differentiating through Optimization Paths

I'm reading the paper "Optimizing Millions of Hyperparameters by Implicit Differentiation". The key contribution of the paper is to show that you can replace optimizing through the ...
Decadz's user avatar
  • 1
0 votes
1 answer
453 views

How to explain near zero gradients on first epochs?

As I understand the gradient should reflect how near the weights are to the optimal values. In this way i will expect that on the first epochs the gradients far from zero or at least not mostly zero ...
Guillermo Alvarez's user avatar
1 vote
0 answers
472 views

Why would one prefer the gradient of the sum rather than the sum of the gradients?

When gradients are aggregated over mini batches, I sometimes see formulations like this, e.g., in the "Deep Learning" book by Goodfellow et al. $$\mathbf{g} = \frac{1}{m} \nabla_{\mathbf{w}} ...
Eddie C's user avatar
  • 11
3 votes
1 answer
223 views

In general - is Stochastic Gradient Descent a "superior" algorithm compared to Gradient Descent? [closed]

On a very informal level, if we were to compare the (classical) Gradient Descent Algorithm to the Stochastic Gradient Descent Algorithm, the first thing that comes to mind is: Gradient Descent can be ...
stats_noob's user avatar
0 votes
1 answer
102 views

Unclear fact about difference between Gradient Descent to Stochastic Gradient Decent in wikipedia

From wikipedia page it mentioned: To economize on the computational cost at every iteration, stochastic gradient descent samples a subset of summand functions at every step. This is very effective in ...
JammingThebBits's user avatar
0 votes
1 answer
46 views

Simple Polynomial Gradient Descent algorithm not working

I am trying to implement a simple 2nd order polynomial gradient descent algorithm in Java. It is not converging and becomes unstable. How do I fix it? ...
PentiumPro200's user avatar
2 votes
0 answers
167 views

Watkins' Q(λ) with function approximation: why is gradient not considered when updating eligibility traces for the exploitation phase?

I'm implementing the Watkins' Q(λ) algorithm with function approximation (in 2nd edition of Sutton & Barto). I am very confused about updating the eligibility traces because, at the beginning of ...
Francesco Vignola's user avatar
1 vote
0 answers
58 views

GANs: Why does iterative gradient descent sometimes optimise $\min_G \max_D V(D,G)$ and sometimes $\max_D \min_G V(D,G)$?

For the following minimax equation for generative adversarial networks (GANs), $$\min_G \max_D V(D,G) = \mathbb{E}_{\boldsymbol{x}\sim p_{data}(\boldsymbol{x})}[\log D(\boldsymbol{x})] + \mathbb{E}_{\...
James Ellis's user avatar
8 votes
2 answers
5k views

Why is gradient descent used over the conjugate gradient method?

Based on some preliminary research, the conjugate gradient method is almost exactly the same as gradient descent, except the search direction must be orthogonal to the previous step. From what I've ...
Recessive's user avatar
  • 1,406

1
2 3 4 5