Questions tagged [gradient-descent]

For questions surrounding gradient descent, a method for finding the optimum state of a parameterized function based on another function often called the loss or error function. It iteratively descends the loss surface to the minimum loss by adjusting parameters based on the product of the partial derivatives comprising the gradient and a learning rate.

Filter by
Sorted by
Tagged with
1
vote
1answer
56 views

What can be deduced about the "algorithm" of backpropagation/gradient descent?

On this video Link to video a neurologist starts by saying that we do not know how neurons calculate gradients for backpropagation. At minute 30:39 hes showing faster convergence for "our algorithm"...
2
votes
1answer
5k views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
1
vote
1answer
133 views

How should I update the weights of a neural network, given the gradient?

After watching 3Blue1Brown's tutorial series, and an array of others, I'm attempting to make my own neural network from scratch. So far, I'm able to calculate the gradient for each of the weights and ...
3
votes
2answers
850 views

Why use semi-gradient instead of full gradient in RL problems, when using function approximation?

Semi-gradient methods work well in reinforcement learning, but what is there a reason of not using the true gradient if it can be computed? I tried it on the cart pole problem with a deep Q-network ...
3
votes
2answers
2k views

How to perform gradient checking in a neural network with batch normalization?

I have implemented a neural network (NN) using python and numpy only for learning purposes. I have already coded learning rate, momentum, and L1/L2 regularization and checked the implementation with ...
3
votes
0answers
729 views

How to calculate gradient of filter in convolution network

I have similar architecture like in image:CNN. I don't understand how to calculate gradient of filter F. I found these equations(source): Gradient and delta, where first equation calculate gradient ...
7
votes
3answers
2k views

CNN backpropagation with stride>1

I read that to compute the derivative of the error with respect to the input of a convolution layer is the same to make of a convolution between deltas of the next layer and the weight matrix rotated ...
2
votes
2answers
789 views

How is direction of weight change determined by Gradient Descent algorithm

The result of gradient descent algorithm is a vector. So how does this algorithm decide the direction for weight change? We Give hyperparameters for step size. But how is the vector direction for ...
7
votes
1answer
7k views

How is the gradient calculated for the middle layer's weights?

I am trying to understand backpropagation. I used a simple neural network with one input $x$, one hidden layer $h$ and one output layer $y$, with weight $w_1$ connecting $x$ to $h$, and $w_2$ ...
1
vote
0answers
48 views

How to calculate Adaptive gradient?

In the FaceNet paper there mentions an gradient algorithm called 'AdaGrad'(Adaptive Gradient) referenced to this paper which has apparently been used to calculate the gradient of the Triplet Loss ...
4
votes
1answer
125 views

What are some concrete steps to deal with the vanishing gradient problem?

I am training an ANN for classification between 3 classes. The ANN has an input layer, one hidden layer and a 3 node output layer. The problem I am facing is that the output being produced by the 3 ...
0
votes
2answers
356 views

What is the proof behind the gradient of a curve being proportional to the distance between the two co-ordinates in the x-axis?

In the [delta rule][1] the equation to adjust the weight with respect to error is $$w_{(n+1)}=w_{(n)}-\alpha \times \frac{\partial E}{\partial w}$$ *where $\alpha$ is the learning rate and $E$ is the ...
10
votes
1answer
990 views

What are the best known gradient-free training methods for deep learning?

As I know, the current state of the art methods for training deep learning networks are variants of gradient descent or stochastic gradient descent. What are the best known gradient-free training ...
11
votes
2answers
4k views

Is the mean-squared error always convex in the context of neural networks?

Multiple resources I referred to mention that MSE is great because it's convex. But I don't get how, especially in the context of neural networks. Let's say we have the following: $X$: training ...
4
votes
1answer
158 views

Can a neural network learn to avoid wrong decisions using backpropagation?

I studied the articles on Neural Networks and Deep Learning from Michael Nielsen and developed a simple neural network based on his examples. I understand how backpropagation works and I already ...
2
votes
0answers
64 views

Are gradients of weights in RNNs dependent on the gradient of every neuron in that layer?

I am writing my own recurrent neural network in Java to understand the inner workings better. While working through the math, I found that in timesteps later than 2 the gradient of weight w of neuron ...
1
vote
2answers
2k views

How would 1D gradient descent look like?

We have always known that gradient descent is a function of two or more variables. But how can we geometrically represent gradient descent if it is a function of only one variable?
1
vote
1answer
71 views

Can a second network take as input the weights of a first network and help training the first network?

I understand that as a network learns about an output with regards to an input, weights are updated according to how wrong the guess was for that node. So, over time, the weights move in the "...
3
votes
1answer
354 views

Does Musk know what gradient descent is?

From Futurism.com: Musk indicates that internet infrastructure is "particularly susceptible" to a method called gradient descent algorithm, a mathematical problem-solving process. Bad news is, AI ...
6
votes
1answer
4k views

How to avoid falling into the "local minima" trap?

How do I avoid my gradient descent algorithm into falling into the "local minima" trap while backpropogating on my neural network? Are there any methods which help me avoid it?

1 2 3
4