For questions surrounding gradient descent, a method for finding the optimum state of a parameterized function based on another function often called the loss or error function. It iteratively descends the loss surface to the minimum loss by adjusting parameters based on the product of the partial derivatives comprising the gradient and a learning rate.

205 questions
Filter by
Sorted by
Tagged with
1 vote
28 views

### Why same learning rate for slope and intercept not working in Linear regression?

I'm a new student in AI, currently learning linear regression. I used the california housing dataset for doing my experiments. My goal is to predict the 'population' column based on the 'total_rooms' ...
1 vote
69 views

### Is there a theoretical way to determine the best learning rate for gradient descent if the function is a simple known polynomial?

I was playing around gradient descent topic. Wrote a function that calculates a gradient descent of a degree-2 polynomial. While trying out what is the best "step size multiplyer" (a.k.a. &...
• 113
126 views

• 121
540 views

### In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?

Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one ...
1 vote
572 views

### What is the effect of gradient clipping by norm on the performance of a model?

It is recommended to apply gradient clipping by normalization in case of exploding gradients. The following quote is taken from here answer One way to assure it is exploding gradients is if the loss ...
• 3,830
1 vote
259 views

### What is the difference between gradient decent in neural networks and temporal difference in reinforcement learning?

I am studying Q-learning in reinforcement learning. My question is about the Bellman equation. In Q-learning, the Bellman equation is often introduced as follows. \begin{align} Q_{new}(s,a) &= Q_{...
1 vote