Questions tagged [gradient-descent]

For questions surrounding gradient descent, a method for finding the optimum state of a parameterized function based on another function often called the loss or error function. It iteratively descends the loss surface to the minimum loss by adjusting parameters based on the product of the partial derivatives comprising the gradient and a learning rate.

Filter by
Sorted by
Tagged with
1
vote
1answer
57 views

Different ways to calculate backpropagation derivatives, any difference?

I'm studying error backpropagation in neural networks. I am interested in why we use only one path on the computational graph to get the value of the derivative for a weight? I ask the question ...
1
vote
1answer
25 views

Do gradient-based algorithms deal with the flat regions with desired points?

I am studying a chapter named Numerical Computation of a deep learning book. Afaik, it does not deal with flat regions with desired points. For example, let us consider a function whose local/global ...
0
votes
0answers
24 views

Is there any significance for higher order gradients in artificial intelligence?

Although I don't know in detail, I am aware of the following facts regarding the usefulness of gradients in some domains of artificial intelligence, especially in optimization. First order gradient: ...
0
votes
2answers
29 views

Reason for relaxing limit in derivative in this context?

Consider the following paragraph from NUMERICAL COMPUTATION of the deep learning book.. Suppose we have a function $y = f(x)$, where both $x$ and $y$ are real numbers. The derivative of this function ...
0
votes
0answers
11 views

What is meant by non-convergent limit cycles?

Limit cycle is a closed curve that is isolated i.e., no other closed curve near to it. You can read the following paragraph from here If there is (such) a closed curve, the nearby trajectories must ...
0
votes
0answers
31 views

Is there any geometrical interpretation on overcoming gradient related problems by adjusting/changing loss function?

There are instances in literature where we need to change loss function in order to escape from gradient problems. Let $L_f$ be a loss function for a model I need to train on. Some times $L_f$ leads ...
1
vote
1answer
53 views

Why is it called "batch" gradient descent if it consumes the full dataset before calculating the gradient?

While training a neural network, we can follow three methods: batch gradient descent, mini-batch gradient descent and stochastic gradient descent. For this question, assume that your dataset has $n$ ...
3
votes
1answer
55 views

What are the necessary mathematical properties to be a loss function in gradient based optimizations?

Loss functions are used in training neural networks. I am interested in knowing the mathematical properties that are necessary for a loss function to participate in gradient descent optimization. I ...
2
votes
1answer
94 views

In logistic regression, why is the binary cross-entropy loss function convex?

I am studying logistic regression for binary classification. The loss function used is cross-entropy. For a given input $x$, if our model outputs $\hat{y}$ instead of $y$, the loss is given by $$\text{...
0
votes
1answer
86 views

Why is loss displayed as a parabola in mean squared error with gradient descent?

I'm looking at the loss function: mean squared error with gradient descent in machine learning. I'm building a single-neuron network (perceptron) that outputs a linear number. For example: Input * ...
1
vote
0answers
33 views

Could the inputs of the mean squared-error loss function be transformed to allow larger learning rates?

In the context of a neural network $\hat{y} = f_\theta(\mathbf{x})$ with parameters $\theta$ that is trained to perform regression such that the prediction $\hat{\mathbf{y}} = [\hat{y}_1,\hat{y}_2,...,...
-1
votes
1answer
31 views

How can the gradient of the weight be calculated in the viewpoint of matrix calculus?

Let $\sigma(x)$ be sigmoid function. Consider the case where $\text{out}=\sigma(\vec{x} \times W + \vec{b})$, and we want to compute $\frac{\partial{\text{out}}}{\partial{w} }.$ Set the dimension as ...
5
votes
3answers
212 views

Does gradient descent in deep learning assume a smooth fitness landscape?

I've come across the concept of fitness landscape before and, in my understanding, a smooth fitness landscape is one where the algorithm can converge on the global optimum through incremental ...
1
vote
0answers
34 views

Optimizer that prevents parameters from oscillating

When we perform gradient descent, especially in an online setting where the training data is presented in a non-random order, a particular 1-dimensional parameter (such as an edge weight) may first ...
0
votes
0answers
25 views

Do Gradient Descent and Natural Gradient solve the same problem?

I am troubled by natural gradient methods. If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do? I found on https://...
0
votes
0answers
17 views

What is the effect and behavior of using mixed weight instead of normal weight matrix?

Suppose I try to find appropriate matrix A in differential equation $\dot{X}=A X$ using RNN. Current state is $X=\begin{bmatrix} x_{1}\\ x_{2}\\ \end{bmatrix}$, and desired trajectory state is $X_d=\...
1
vote
1answer
68 views

In gradient descent's update rule, why do we use $\sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}$ instead of $\frac{\delta C_0}{\delta w^{l}}$?

I am trying to code a two layered neural network simple NN as I have described here https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/ I am getting stuck on the last step of updating the ...
2
votes
0answers
56 views

How to deal with losses on different scales in multi-task learning?

Say I'm training a model for multiple tasks by trying to minimize sum of losses $L_1 + L_2$ via gradient descent. If these losses are on a different scale, the one whose range is greater will dominate ...
2
votes
0answers
26 views

What is the name of this algorithm that estimates the gradient with an average by sampling from a distribution?

Consider maximizing the function $R(w)$ with parameter $w$ using gradient ascent. However, we don't know the gradient $\nabla_wR(w)$ formula. Now suppose $w$ is sampled from a probability distribution ...
2
votes
0answers
83 views

Bias gradient of layer before batch normalization always zero

From the original paper and this post we have that batch normalization backpropagation can be formulated as I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
7
votes
2answers
336 views

Is there an ideal range of learning rate which always gives a good result almost in all problems?

I once read somewhere that there is a range of learning rate within which learning is optimal in almost all the cases, but I can't find any literature about it. All I could get is the following graph ...
2
votes
0answers
34 views

Should I use batch gradient descent when I have a small sample size?

I have a dataset with an input size of 155x155, with the output being 155 x 1 with a 3-4 layer neural net being used for regression. With such a small sample size, should I use full batch gradient ...
4
votes
0answers
59 views

Why are most commonly used activation functions continuous?

I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with ...
2
votes
0answers
23 views

Is it possible to ensure the convergence when training a RNN weight on its SVD decomposition?

I'm reading the following paper in which the author seems to do 2 things interesting: The hidden-to-hidden weight matrix of the RNN is SVD decomposed and train separately. Each orthogonal part of the ...
0
votes
0answers
33 views

Backpropagation implementation not applicable for other cases

I saw this implementation of backpropagation in MATLAB, where the loss function used is MSE, and the last layer's activation function was sigmoid. I denoted the portions of the formula for what I ...
1
vote
0answers
28 views

How to derive compact convex set K and its diameter D to program Accelegrad algorithm in practice?

Given the original paper (https://arxiv.org/pdf/1809.02864.pdf), I would like to implement the Accelegrad algorithm for which I report the pseudocode of the paper: In the pseudocode, the authors ...
0
votes
1answer
79 views

Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?

If the derivative is supposed to give the rate of change of a function at that point, then why is the derivative of the softmax layer (a vector) the Jacobian matrix, which has a different shape than ...
0
votes
1answer
45 views

What is the derivative of a specific output with respect to a specific weight?

If I have a neural network, and say the 6th output node of the neural network is: $$x_6 = w_{16}y_1 + w_{26}y_2 + w_{36}y_3$$ What does that make the derivative of: $$\frac{\partial x_6}{\partial w_{...
2
votes
2answers
85 views

Is there anything that ensures that convolutional filters end up different from one another?

I found this question very interesting, and this is a follow up on it. Presumably, we'd want all the filters to converge towards some complementary set, where each filter fills as large a niche as ...
0
votes
1answer
40 views

Loss randomly changing, incorrect output (even for low loss) when trying to overfit on a single set of input and output

I am trying to make a neural network framework from scratch in C++ just for fun, and to test my backpropagation, I thought it would be an easy way to test the functionality if I give it one input - a ...
6
votes
1answer
110 views

During neural network training, can gradients leak sensitive information in case training data fed is encrypted (homomorphic)?

Some algorithms in the literature allow recovering the input data used to train a neural network. This is done using the gradients (updates) of weights, such as in Deep Leakage from Gradients (2019) ...
0
votes
1answer
54 views

RMSprop equation - dividing by a matrix?

I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me. $dW$ and $db$ are matrices (that's what I understand from the element-wise comment), so that must ...
1
vote
0answers
205 views

What is the gradient of an attention unit?

The paper Attention Is All You Need describes the Transformer architecture, which describes attention as a function of the queries $Q = x W^Q$, keys $K = x W^K$, and values $V = x W^V$: $\text{...
1
vote
0answers
37 views

How to use unmodified input in neural network?

My question may be a bit hard to explain... My neural network learns a categorical distribution, which serves as an index. This index will look up the value (= action_mean) in Input 2. From this ...
2
votes
0answers
66 views

Why should the weight updates be proportional to input?

I'm reading the book Grokking Deep Learning. Regarding weight updates during training, it has the following code and explanation: ...
2
votes
1answer
368 views

What is the impact of scaling the KL divergence and reconstruction loss in the VAE objective function?

Variational autoencoders have two components in their loss function. The first component is the reconstruction loss, which for image data, is the pixel-wise difference between the input image and ...
8
votes
2answers
217 views

Why is the perceptron criterion function differentiable?

I'm reading chapter one of the book called Neural Networks and Deep Learning from Aggarwal. In section 1.2.1.1 of the book, I'm learning about the perceptron. One thing that book says is, if we use ...
1
vote
1answer
44 views

How does vanish gradient restrict RNN to not work for long range dependencies?

I am really trying to understand deep learning models like RNN, LSTMs etc. I have gone through many tutorials of RNN and have learned that RNN cannot work for long Range dependencies, like: Consider ...
0
votes
0answers
24 views

Random Initializations with ReLU gives puzzling results

this may sound naive, but I’m getting a really puzzling result. I was experimenting with MNIST on vanilla MLP (784, 256, 128, 10) with ...
0
votes
0answers
34 views

Bigger models get higher losses

I'm training a model with the transformer encoder architecture on a given fixed set of data. The task I'm solving has a trivial approximation which consists in copying part of the input to the output, ...
4
votes
2answers
534 views

What is the advantage of using cross entropy loss & softmax?

I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found & fixed some ...
3
votes
0answers
96 views

What is the time complexity for training a gated recurrent unit (GRU) neural network using back-propagation through time?

Let us assume we have a GRU network containing $H$ layers to process a training dataset with $K$ tuples, $I$ features, and $H_i$ nodes in each layer. I have a pretty basic idea how the complexity of ...
2
votes
1answer
48 views

Why is the fraction of time spent in state $s$, $\mu(s)$, not in the update rule of the parameters?

I am reading "Reinforcement Learning: An Introduction (2nd edition)" authored by Sutton and Barto. In Section 9, On-policy prediction with approximation, it first gives the mean squared ...
1
vote
0answers
27 views

Why scaling down the parameter many times during training will help the learning speed be the same for all weights in Progressive GAN?

The title is one of the special things in Progressive GAN, a paper of the NVIDIA team. By using this method, they introduced that Our approach ensures that the dynamic range, and thus the learning ...
0
votes
1answer
55 views

Flatten image using Neural network and matrix transpose

I have read a lecture note of Prof. Andrew Ng. There was something about data normalization like how can we flatten an image of (64x64x3) into a (64x64x3)*x1 vector. After that there is pictorial ...
3
votes
1answer
106 views

Why is the learning rate generally beneath 1?

In all examples I've ever seen, the learning rate of an optimisation method is always less than $1$. However, I've never found an explanation as to why this is. In addition to that, there are some ...
1
vote
1answer
57 views

How parameter adjustment works in Gradient Descent?

I am trying to comprehend how the Gradient Descent works. I understand we have a cost function which is defined in terms of the following parameters, $J(𝑤_{1},𝑤_{2},.... , w_{n}, b)$ the derivative ...
4
votes
2answers
147 views

What is the goal of weight initialization in neural networks?

This is a simple question. I know the weights in a neural network can be initialized in many different ways like: random uniform distribution, normal distribution, and Xavier initialization. But what ...
6
votes
1answer
224 views

What, exactly, does the REINFORCE update equation mean?

I understand that this is the update for the parameters of a policy in REINFORCE: $$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t} $$ Where 𝑣𝑡 is ...
0
votes
1answer
46 views

What are the rules behind vector product in gradient?

Let's suppose we have calculated the gradient and it came out to be $f(WX)(1-f(W X))X$, where $f()$ is the sigmoid function, $W$ of order $2\times2$ is the weight matrix, and $X$ is an input vector of ...