Questions tagged [gradient-descent]

For questions surrounding gradient descent, a method for finding the optimum state of a parameterized function based on another function often called the loss or error function. It iteratively descends the loss surface to the minimum loss by adjusting parameters based on the product of the partial derivatives comprising the gradient and a learning rate.

Filter by
Sorted by
Tagged with
1
vote
1answer
22 views

How does vanish gradient restrict RNN to not work for long range dependencies?

I am really trying to understand deep learning models like RNN, LSTMs etc. I have gone through many tutorials of RNN and have learned that RNN cannot work for long Range dependencies, like: Consider ...
0
votes
0answers
19 views

Random Initializations with ReLU gives puzzling results

this may sound naive, but I’m getting a really puzzling result. I was experimenting with MNIST on vanilla MLP (784, 256, 128, 10) with ...
0
votes
0answers
30 views

Bigger models get higher losses

I'm training a model with the transformer encoder architecture on a given fixed set of data. The task I'm solving has a trivial approximation which consists in copying part of the input to the output, ...
3
votes
2answers
66 views

What is the advantage of using cross entropy loss & softmax?

I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found & fixed some ...
3
votes
0answers
45 views

What is the time complexity for training a gated recurrent unit (GRU) neural network using back-propagation through time?

Let us assume we have a GRU network containing $H$ layers to process a training dataset with $K$ tuples, $I$ features, and $H_i$ nodes in each layer. I have a pretty basic idea how the complexity of ...
2
votes
1answer
36 views

Why is the fraction of time spent in state $s$, $\mu(s)$, not in the update rule of the parameters?

I am reading "Reinforcement Learning: An Introduction (2nd edition)" authored by Sutton and Barto. In Section 9, On-policy prediction with approximation, it first gives the mean squared ...
1
vote
0answers
25 views

Why scaling down the parameter many times during training will help the learning speed be the same for all weights in Progressive GAN?

The title is one of the special things in Progressive GAN, a paper of the NVIDIA team. By using this method, they introduced that Our approach ensures that the dynamic range, and thus the learning ...
0
votes
1answer
20 views

Flatten image using Neural network and matrix transpose

I have read a lecture note of Prof. Andrew Ng. There was something about data normalization like how can we flatten an image of (64x64x3) into a (64x64x3)*x1 vector. After that there is pictorial ...
1
vote
1answer
38 views

How parameter adjustment works in Gradient Descent?

I am trying to comprehend how the Gradient Descent works. I understand we have a cost function which is defined in terms of the following parameters, $J(𝑤_{1},𝑤_{2},.... , w_{n}, b)$ the derivative ...
4
votes
2answers
71 views

What is the goal of weight initialization in neural networks?

This is a simple question. I know the weights in a neural network can be initialized in many different ways like: random uniform distribution, normal distribution, and Xavier initialization. But what ...
5
votes
1answer
110 views

What, exactly, does the REINFORCE update equation mean?

I understand that this is the update for the parameters of a policy in REINFORCE: $$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t} $$ Where 𝑣𝑡 is ...
0
votes
0answers
32 views

linear regression using gradient decent

I am novice in machine learning and try to implement Linear regression algorithm using Gradient Descent. I am using Google Co-...
0
votes
1answer
39 views

What are the rules behind vector product in gradient?

Let's suppose we have calculated the gradient and it came out to be $f(WX)(1-f(W X))X$, where $f()$ is the sigmoid function, $W$ of order $2\times2$ is the weight matrix, and $X$ is an input vector of ...
2
votes
1answer
26 views

Is batch learning with gradient descent equivalent to “rehearsal” in incremental learning?

I am learning about incremental learning and read that rehearsal learning is retraining with old data. In essence, isn't this the exact same thing as batch learning (with stochastic gradient descent)? ...
1
vote
1answer
43 views

Implementing Gradient Descent Algorithm in Python, bit confused regarding equations

I'm following the guide as outlined at this link: http://neuralnetworksanddeeplearning.com/chap2.html For the purposes of this question, I've written a basic network 2 hidden layers, one with 2 ...
0
votes
1answer
63 views

Is my understanding of back-propogation correct?

I am trying to learn backpropagation and this is what I know so far. To update the weights of the neural network you have to figure out the partial derivative of each of the parameters on the loss ...
1
vote
0answers
33 views

Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient?

Isn't it true that using max over a softmax will be much slower because there is not a smooth gradient? Max basically zeros out the gradients of all the non-maximum values. Especially at the beginning ...
2
votes
1answer
52 views

If the normal equation works, why do we need gradient descent?

Recently, I followed the open course CS229, http://cs229.stanford.edu/notes/cs229-notes1.pdf This lecturer introduces an alternative approach to gradient descent that is called "Normal Equation&...
1
vote
2answers
73 views

Why the cost/loss starts to increase for some iterations during the training phase?

I am trying to build a recurrent neural network from scratch. It's a very simple model. I am trying to train it to predict two words (dogs and gods). While training, the value of cost function starts ...
0
votes
1answer
316 views

What is the relationship between gradient accumulation and batch size?

I am currently training some models using gradient accumulation since the model batches do not fit in GPU memory. Since I am using gradient accumulation, I had to tweak the training configuration a ...
1
vote
0answers
27 views

What is the equation of the learning rate decay in the Adam optimiser?

Adam is known as an algorithm that has an adaptive learning rate for each parameter. I believe this is due to the division by the term $$v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot g_t^2 $$ Hence, ...
3
votes
0answers
60 views

How does SGD escape local minima?

SGD is able to jump out of local minima that would otherwise trap BGD I don't really understand the above statement. Could someone please provide a mathematical explanation for why SGD (Stochastic ...
1
vote
0answers
33 views

If the output of a model is a ridge function, what should the activation functions at all the nodes be?

I have the following assignment. I can't understand the b part of this question in my assignment. I have completed the 1st part and understand the maths behind it, but the 2nd part has me stumped. I ...
4
votes
1answer
70 views

How is the Jacobian a generalisation of the gradient?

I came across these slides Natural Language Processing with Deep Learning CS224N/Ling284, in the context of natural language processing, which talk about the Jacobian as a generalization of the ...
2
votes
1answer
108 views

How long should the state-dependent baseline for policy gradient methods be trained at each iteration?

How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods? I'm using this equation to ...
0
votes
0answers
89 views

Backward pass of CNN like Resnet: how to manually compute flops during backprop?

I've been trying to figure out how to compute the number of Flops in backward pass of ResNet. For forward pass, it seems straightforward: apply the conv filters to the input for each layer. But how ...
0
votes
0answers
31 views

How does multiple outputs added from intermediate layers of a keras functional model influence its learning behaviour / gradients?

let us assume I have a keras functional model with 2 inputs. My model has two branchs, each branch for each input. The model only uses dense layers. The first input is the data itself (feature vector ...
3
votes
2answers
145 views

What exactly is averaged when doing batch gradient descent?

I have a question about how the averaging works when doing mini-batch gradient descent. I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-...
2
votes
1answer
75 views

Why do we update all layers simultaneously while training a neural network?

Very deep models involve the composition of several functions or layers. The gradient tells how to update each parameter, under the assumption that the other layers do not change. In practice, we ...
1
vote
0answers
59 views

What is the gradient of a non-linear SVM with respect to the input?

The objective function of an SVM is the following: $$J(\mathbf{w}, b)=C \sum_{i=1}^{m} \max \left(0,1-y^{(i)}\left(\mathbf{w}^{t} \cdot \mathbf{x}^{(i)}+b\right)\right)+\frac{1}{2} \mathbf{w}^{t} \...
1
vote
1answer
48 views

What do these numbers represent in this picture of a surface?

The following image is a screenshot from a video tutorial that illustrates the concept of gradient descent algorithm with a 3D animation. Do the numbers on the top of the balls pointed out by the red ...
3
votes
0answers
86 views

How to implement Mean square error loss function in mini batch GD

I have a vectorized implementation of the neural network in c++. I successfully solve the classification problems of Fashion MNIST and CIFAR. Now I am modifying my code to do the Linear regression. I ...
0
votes
0answers
34 views

What is the degree of linearity in the error propagated by Gradient Descent?

Neural Network is trained to learn a non-linear function, the more layers it has, the more is the quality of the prediction and the ability to match the real-world function correctly (lets leave aside ...
1
vote
1answer
51 views

What is relation between gradient descent and regularization in deep learning?

Gradient descent is used to reduce the loss and regularization is used to fight over-fitting. Is there any relation between gradient descent and regularization, or both are independent of each other?...
2
votes
1answer
80 views

Different methods of calculating gradients of cost function(loss function)

We require to find the gradient of loss function(cost function) w.r.t to the weights to use optimization methods such as SGD or gradient descent. So far, I have come across two ways to compute the ...
1
vote
1answer
32 views

How are the weights retained for filters for a particular class in a CNN?

I am new to CNN. What I have learned so far about the filters is that when we are giving a training example to our model, our model updates the weights by gradient descent to minimize the loss ...
1
vote
1answer
30 views

What is the difference between batch and mini-batch gradient decent?

I am learning deep learning from Andrew Ng's tutorial Mini-batch Gradient Descent. Can anyone explain the similarities and dissimilarities between batch GD and mini-batch GD?
1
vote
0answers
44 views

Which activation functions can lead to the vanishing gradient problem?

From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem. What other activation functions can lead to the ...
1
vote
3answers
103 views

How to prove that gradient descent doesn't necessarily find the global optimum?

How can I prove that gradient descent doesn't necessarily find the global optimum? For example, consider the following function $$f(x_1, x_2, x_3, x_4) = (x_1 + 10x_2)^2 + 5x_2^3 + (x_2 + 2x_3)^4 + ...
3
votes
1answer
72 views

What is the purpose of argmax in the PPO algorithm?

I'm kinda new to machine learning and still not too solid on math and particularly calculus. I'm currently trying to implement PPO algorithm as described in the spiningUp website : This line is ...
0
votes
1answer
64 views

How do you perform a gradient based adversarial attack on an SVM based model?

I have an SVM currently and want to perform a gradient based attack on it similar to FGSM discussed in Explaining And Harnessing Adversarial Examples. I am struggling to actually calculate the ...
1
vote
1answer
127 views

Understanding the derivation of the first-order model-agnostic meta-learning

According to the authors of this paper, to improve the performance, they decided to drop backward pass and using a first-order approximation I found a blog which discussed how to derive the math ...
1
vote
1answer
63 views

Oscillating around the saddle point in gradient descent?

I was reading a blog post that talked about the problem of the saddle point in training. In the post, it says if the loss function is flatter in the direction of x (local minima here) compared to ...
2
votes
0answers
64 views

Why gradients are so small in deep learning?

The learning rate in my model is 0.00001 and the gradients of the model is within the distribution of [-0.0001, 0.0001]. Is it ...
0
votes
0answers
13 views

How to perform PGD on a pretrained CNN?

I have a pretrained CNN model using the keras library. I now need to perform a Projected Gradient Descent (PGD) to develop some adversarial examples. To do this, I will need to perform a gradient ...
4
votes
1answer
81 views

Is there a reason to choose regular momentum over Nesterov momentum for neural networks?

I've been reading about Nesterov momentum from here and it seems like a nice improvement over regular momentum with no extra cost whatsoever. However, is this really the case? Are there instances ...
1
vote
0answers
52 views

Is Gradient Descent algorithm a part of Calculus of Variations?

As in https://en.wikipedia.org/wiki/Calculus_of_variations The calculus of variations is a field of mathematical analysis that uses variations, which are small changes in functions and ...
0
votes
0answers
29 views

How would I go about performing a single step of gradient descent on this model?

I have a classification model that consists of a CNN followed by an SVM. I used the Keras library for the CNN portion and sklearn for the SVM portion. I am assuming I will have to fiddle with the ...
3
votes
3answers
371 views

What's the function that SGD takes to calculate the gradient?

I'm struggling to fully understand the stochastic gradient descent algorithm. I know that gradient descent allows you to find the local minimum of a function. What I don't know is what exactly that ...
6
votes
1answer
625 views

What is the formula for the momentum and Adam optimisers?

In the gradient descent algorithm, the formula to update the weight $w$, which has $g$ as the partial gradient of the loss function with respect to it, is: $$w\ -= r \times g$$ where $r$ is the ...