Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gradient-descent]

For questions surrounding gradient descent, a method for finding the optimum state of a parameterized function based on another function often called the loss or error function. It iteratively descends the loss surface to the minimum loss by adjusting parameters based on the product of the partial ...

1
vote
1answer
28 views

Weight Normalization paper

I am trying to dissect paper about weight normalization: https://papers.nips.cc/paper/6114-weight-normalization-a-simple-reparameterization-to-accelerate-training-of-deep-neural-networks.pdf ...
1
vote
0answers
12 views

Is anyone extending torch.optim.Optimizer for Nesterov-accelerated and non-constent β ADAM?

Varieties of stochastic gradient descent are currently dominant in deep learning engineering. Training efficiency can be optimized by beginning with a high learning rate and tapering it to a lower ...
4
votes
1answer
69 views

Do good approximations produce good gradients?

Let’s say I have a neural net doing classification and I’m doing stochastic gradient descent to train it. If I know that my current approximation is a decent approximation, can I conclude that my ...
0
votes
0answers
26 views

Correct implementation of backpropagation / gradient calculation for a single kernel convolutional neural network

I have coded a convolutional neuronal network with just one filter/kernel which slides over a word embedding with a dimension of 100. With a kernel size of 4x1, I got a feature map $m$ with the size ...
4
votes
1answer
68 views

Can non-differentiable layer be used in a neural network, if it's not learned?

For example, AFAIK pooling layer in CNN is not differentiable, but it can be used because it's not learning. Is it always true?
4
votes
1answer
47 views

TRPO/PPO importance sampling term in loss function

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients ...
1
vote
0answers
16 views

Has anyone investigated iteration awareness beyond RNN and LSTM?

This question considers the convergence of an artificial networks (MLPs, RNNs, LSTM nets, CNNs) over time or over the course of epochs made up of iterations through training examples. In this ...
1
vote
2answers
69 views

Mini-batch training and the gradient

I am reading a book that states, "As the mini-batch size increases, the gradient computed is closer to the 'true' gradient". So basically what they are saying is mini-batch training only focuses on ...
0
votes
2answers
43 views

Behaviour of cost

In doing a project using neural networks with an input layer, 4 hidden layers and an output layer ,I used mini batch gradient descent. I noticed that the randomly initialised weights seemed to do a ...
5
votes
2answers
90 views

Working with large datasets

If one has a dataset large enough to learn a highly complex function, say learning chess game-play, and the processing time to run mini batch gradient descent on this entire dataset is too high, can I ...
2
votes
0answers
25 views

Gradient of boltzmann policy over reward function

I'm struggling with an inverse reinforcement learning problem which seems to appear quite often around the literature, yet I can't find any resources explaining it. The problem is that of calculating ...
1
vote
1answer
31 views

Gradient descent update

In the following, I put the link for the general algorithm of maximum entropy inverse reinforcement learning. http://178.79.149.207/assets/maxent/maxent_slide.jpg This uses a gradient descent ...
2
votes
1answer
89 views

Why Feature Scaling for skewed contour?

Why is it that the skewed contour (unscaled features) will result in slow performance of gradient descent? In other words, how (or why) will the gradients end up taking a long time before finding the ...
1
vote
1answer
38 views

What can be deduced about the “algorithm” of backpropagation/gradient descent?

On this video Link to video a neurologist starts by saying that we do not know how neurons calculate gradients for backpropagation. At minute 30:39 hes showing faster convergence for "our algorithm"...
1
vote
1answer
91 views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
0
votes
1answer
27 views

How to change a weight/bias with gradient

After watching 3Blue1Brown's tutorial series, and an array of others, I'm attempting to make my own neural network from scratch. So far, I'm able to calculate the gradient for each of the weights and ...
1
vote
1answer
119 views

Why use semi-gradient instead of full gradient in RL problems, when using function approximation?

Semi gradient methods work well in Reinforcement Learning, but what is the reason of not using the true gradient if it can be computed? I tried it on the cart pole problem with a deep Q-Network and it ...
3
votes
2answers
189 views

How to perform gradient checking in a neural network with batch normalization?

I have implemented a neural network (NN) using python and numpy only for learning purposes. I have already coded learning rate, momentum, and L1/L2 regularization and checked the implementation with ...
2
votes
0answers
152 views

How to calculate gradient of filter in convolution network

I have similar architecture like in image:CNN. I don't understand how to calculate gradient of filter F. I found these equations(source): Gradient and delta, where first equation calculate gradient ...
4
votes
1answer
218 views

CNN backpropagation with stride>1

I read that to compute the derivative of the error respect to the input of a convolution layeris the same to make of a convolution between deltas of the next layer and the weight matrix rotated by $...
2
votes
2answers
94 views

How is direction of weight change determined by Gradient Descent algorithm

The result of gradient descent algorithm is a vector. So how does this algorithm decide the direction for weight change? We Give hyperparameters for step size. But how is the vector direction for ...
5
votes
2answers
1k views

How is gradient calculated for middle layer weights?

I am trying to understand backpropagation. I used a simple neural network with one input x, one hidden layer h and one output layer y, with weight w1 connecting x to h, and w2 connecting h to y. x--...
1
vote
0answers
30 views

How to calculate Adaptive gradient?

In the FaceNet paper there mentions an gradient algorithm called 'AdaGrad'(Adaptive Gradient) referenced to this paper which has apparently been used to calculate the gradient of the Triplet Loss ...
5
votes
1answer
83 views

What are some concrete steps to deal with the vanishing gradient problem?

So I am training an ANN for classification between 3 classes. The ANN has an input layer, one hidden layer and a 3 node output layer. The problem I am facing is that the output being produced by the ...
0
votes
2answers
98 views

what is the proof behind the gradient of a curve being equal/proportional to the distance between the two co-ordinates in the x-axis [closed]

In the delta rule the equation to adjust the weight with respect to error is :- where is the Learning Rate and E is the ...
5
votes
1answer
237 views

Gradient free training methods for deep learning

As I know, the current state of the art methods for training deep learning networks are variants of gradient descent / stochastic gradient descent. What are the best known gradient-free training ...
7
votes
2answers
237 views

Convexity of MSE in Neural Networks?

I am getting confused reading online about Gradient Descent, Convex and Non Convex Loss functions. Multiple resources I referred to mention that MSE is great because its convex. But I don't get how, ...
4
votes
1answer
120 views

Can a neural network learn to avoid wrong decisions using backpropagation?

I studied the articles on Neural Networks and Deep Learning from Michael Nielsen and developed a simple neural network based on his examples. I understand how backpropagation works and I already ...
2
votes
0answers
33 views

Are gradients of weights in RNNs dependent on the gradient of every neuron in that layer?

I am writing my own recurrent neural network in Java to understand the inner workings better. While working through the math, I found that in timesteps later than 2 the gradient of weight w of neuron ...
-1
votes
1answer
175 views

How would 1D gradient descent look like?

We have always known that gradient descent is a function of two or more variables. But how can we geometrically represent gradient descent if it is a function of only one variable?
-1
votes
1answer
241 views

Does Musk knows what Gradient descent is

I read a tweet from Elon Musk where describes Gradient descent as an evil action that AI are good at, despite the fact that it is just one of the old, inflexible and not-so-efficient error correction ...
4
votes
1answer
832 views

How to avoid falling into the “local minima” trap? [closed]

How do I avoid my gradient descent algorithm into falling into the "local minima" trap while backpropogating on my neural network? Are there any methods which help me avoid it?