For questions surrounding gradient descent, a method for finding the optimum state of a parameterized function based on another function often called the loss or error function. It iteratively descends the loss surface to the minimum loss by adjusting parameters based on the product of the partial derivatives comprising the gradient and a learning rate.

214 questions
Filter by
Sorted by
Tagged with
36 views

### How is this z-loss implementation in t5x related to this paper's loss X?

I was looking into the loss function in t5x here and see there is a z-loss added to the typical log loss definition. The only paper I could surface on this was https://arxiv.org/abs/1604.08859, but I ...
• 247
13 views

### Nesterov Accelerated Gradient is Slower than Momentum [closed]

I'm working on really simple FFNN on Fashion MNIST to test out different optimizers by implementing them from scratch in NumPy. I'm finding that Nesterov Accelerated Gradient is slower than regular ...
• 115
29 views

### Different Definitions of Momentum -- which one should I work with?

I'm seeing different manners to define momentum, I'm not sure if there is significant difference or not. From my thinking, they seem to do a similar thing mathematically and in practice but I'm ...
• 115
33 views

### Learning Rate greater than ~0.00005 significantly hinders model performance and increases loss

I have been trying to train a model with 0.001 learning rate. I tried regression techniques, early stopping and lr manipulations within epochs. But nothing felt right even though after numerous tries ...
40 views

### custom neural network implementation is giving 10% accuracy on mnist dataset

I've created a toy neural network in python for learning purposes and decided to test it on mnist dataset, I didn't expect great results but the result that I got - 10% is as good as a guess. For many ...
25 views

### Not Averaging Gamma and Beta Gradients in BatchNormalization leads me to higher accuracy

I'm implementing batchnorm from scratch in pure NumPy. I noticed something interesting. While I'm calculating the gradients of gamma (dg) and beta (db), ignoring the summation / averaging of the ...
• 115
17 views

### Covering missing derivations in Bengio RNN exploding gradient/output paper

While I have no doubts about the soundness of its conclusions given that it is very well-known, I have been looking through the classic paper on RNNs which is used to substantiate the claims that ...
• 101
11 views

### collaborative filtering using linear regression

Currently doing andrew ng's unsupervised learning specialization, I came across this algorithm for collaborative filtering: here the Xi refers to feature vector of objects(ex: action in movies, ...
22 views

### Is there any purpose of altering neural network architecture if validation loss does not decrease but training loss does?

I am training a transformer based neural network and the validation loss is not decreasing, but the training loss does decrease. I am wondering if it's possible to debug or change the architecture ...
• 213
1 vote
35 views

### Why same learning rate for slope and intercept not working in Linear regression?

I'm a new student in AI, currently learning linear regression. I used the california housing dataset for doing my experiments. My goal is to predict the 'population' column based on the 'total_rooms' ...
1 vote
73 views

### Is there a theoretical way to determine the best learning rate for gradient descent if the function is a simple known polynomial?

I was playing around gradient descent topic. Wrote a function that calculates a gradient descent of a degree-2 polynomial. While trying out what is the best "step size multiplyer" (a.k.a. &...
• 113
150 views

• 41
1 vote
334 views

### Why do we use gradient descent to minimize the loss function?

The purpose of training neural networks is to minimize a loss function, in this process we usually use gradient descent method. But in Calculus, if we want to find the global minimum of a ...
• 111
216 views

### What exactly is the AI explainability problem?

I am pretty new to AI and have recently been paying attention to AI explainability and the fact that it remains a hurdle within the path of commercializing certain AI systems in health for instance. I ...
1 vote
666 views

### How does MAML inner loop optimization works?

I started to learn meta-learning, reading the MAML paper https://arxiv.org/pdf/1703.03400.pdf In the inner loop, I am calculating adapted parameters for each task, I will be doing multiple steps of ...
• 43
1 vote
98 views

### At which point, does the momentum based GD helps really in this figure?

Classical gradient descent algorithms sometimes overshoot and escape minima as they depend on the gradient only. You can see such a problem during the update from point 6. In classical GD algorithm, ...
• 3,930
28 views

### Resolving Derivation Discrepancies for Differentiating through Optimization Paths

I'm reading the paper "Optimizing Millions of Hyperparameters by Implicit Differentiation". The key contribution of the paper is to show that you can replace optimizing through the ...
453 views

### How to explain near zero gradients on first epochs?

As I understand the gradient should reflect how near the weights are to the optimal values. In this way i will expect that on the first epochs the gradients far from zero or at least not mostly zero ...
1 vote