Questions tagged [backpropagation]
For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.
271
questions
0
votes
0
answers
10
views
NaN gradients while Training (but loss isn't NaN neither the computational graph is disconnected)
The reason I am asking this here is that I haven't found a bug in my code and maybe there isn't a bug at all (or maybe there is). I just want to validate the idea that I am trying to implement. Here ...
- 101
1
vote
1
answer
102
views
Why in one article to calcule the delta uses the weights of current layer, while in another article uses the weights of neuron inputs of next layer?
Article 1:
https://pyimagesearch.com/2021/05/06/backpropagation-from-scratch-with-python/
Article 2:
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/
I was ...
- 243
1
vote
0
answers
35
views
Neural network learns to mimic distribution of classes in dataset instead of using signal from input
I'm trying to implement example from a classic AI paper named "Learning representations by back-propagating errors" by Hinton et al.
Example aims at training network able to predict third ...
- 11
0
votes
0
answers
29
views
How was the word2vec model trained?
Let's take the CBOW (continuous bag of words) model as the example.
Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
- 1
0
votes
1
answer
40
views
If I manually trained a multilayer percetron neural network, always following exactly the same steps meticulously, would I always get the same result?
Reference:
https://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html
If I trained a multilayer percetron neural network manually, following exactly the backpropagation steps described in the article, ...
- 243
0
votes
2
answers
116
views
Does this article make use of the chain rule? And where?
References:
Chain rule in Wikipedia: https://en.wikipedia.org/wiki/Chain_rule
Chain rule in Towards Data Science: https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd#:~...
- 243
1
vote
1
answer
45
views
In multilayer perceptron neural networks, are the names "delta", "gradient" and "error" all the same thing? or not?
What is the difference between "delta", "gradient" and "error",
are these names the same thing?
I'm confused because someone once told me that both the names "delta&...
- 243
1
vote
1
answer
64
views
Please, could someone help me understand if the backpropagation explanations in these two articles about calculate the error are equivalent?
I have a question about backpropagation, I'm a beginner, I'm studying the formulas to calculate the delta of neurons, there are several sources on the internet, which teach in different ways, so I'm ...
- 243
0
votes
0
answers
17
views
Does the training time of a neural network increase more if we add a layer at the beginning or at the end?
Let's consider a fixed NN architecture, dataset and hardware. We add a layer, either at the beginning or at the end of the NN. In which case the training time will increase more? Intuitively, I ...
- 194
1
vote
1
answer
359
views
what is the backpropagation formula to calculate delta and update weights?
I'm trying to study how backpropagation works step by step in a MultiLayer Perceptron neural network. I would really like to be able to understand how these calculations work. And I have a specific ...
- 243
0
votes
0
answers
11
views
Bug in backpropagation algorithm
I've been debugging my backpropagation algorithm for nearly a week and I can't seem to find where I went wrong, yet my network doesn't seem to learn correctly. It learns XOR just fine when I use the ...
- 1
0
votes
0
answers
20
views
Independent parameter update in backpropagation
When we calculate the gradient wrt to each paramters, we consider the other parameters remain constant, but the moment their is a change in any of the other parameters, shouldn't all the other changes ...
0
votes
0
answers
18
views
Is it possible to combine SGD with an unsupervised learning approach effectively
Before I undertake quite a large project I would like to clarify whether my idea for training a multi-layer neural network will work. I plan to make an AI that can land a rocket from randomly ...
- 1
0
votes
1
answer
85
views
Confusion over taking gradients in Variational Autoencoders (VAE)
I am confused as to when to hold certain parameters constant in a VAE. I will explain with a concrete example.
We can write $\operatorname{ELBO}(\phi, \theta) = \mathbb{E}_{q_{\phi}(z)}\left[\log
\...
- 23
0
votes
0
answers
39
views
neural network not training correctly
i'm trying to learn how a neural network works. i'm writing the neural network in C for handwritten digit recognition and training it on MNIST dataset. the neural network has an input layer with 28*28 ...
- 9
0
votes
1
answer
207
views
How to do backpropagation with argmax?
I am attempting to utilize two networks: a classifier and a linear network. Based on the output class of the first network, my goal is to retrieve the corresponding value from the linear network using ...
0
votes
1
answer
97
views
Does the output of LLM's affect their neural weights?
When an LLM creates an output, it seemingly has no way to check if its output was valid. Therefore it wouldn't be able to back-propagate any changes to the weights is used to create that output.
...
- 103
1
vote
2
answers
104
views
Backpropagation with CrossEntropy and Softmax, HOW?
Let Zs be the input of the output layer (for example, Z1 is the input of the first neuron in the output layer), Os be the output of the output layer (which are actually the results of applying the ...
- 11
1
vote
0
answers
32
views
Can gradient descent cause loss to increase in some situations?
Is a gradient descent step always supposed to decrease loss? I can think of a situation where it would seem that gradient descent would increase loss but maybe it I am misunderstanding a part of ...
- 11
1
vote
2
answers
56
views
How to apply backpropagation when one layer of the network is a call-only function (no gradient)?
I worked with Feed Forward Neural Network and VAE and understood backpropagation algorithm. Now I build a VAE network, one layer of it is a very complex vector-to-vector function $f(x)$ (a general '...
- 111
0
votes
1
answer
55
views
About the requirement to compute the gradient at layer $l$
I'm trying to understand a line of my note.
Let's say there is a simple feedforward neural network that has $N$ layers, and for a given layer $l$, it has weight $W^l$, and $g^l$ is the gradient to ...
- 111
1
vote
1
answer
121
views
Is there any reference about backpropagation of the Transformer's multi-head layer?
Is there any reference about backpropagation of the Transformer's multi-head layer or multi-head attention (MHA)? I have searched various journals but have not found one yet.
- 23
1
vote
2
answers
39
views
Is there a resource that offers a detailed overview of the gradient flow?
Understanding the concept of "Gradient Flow" can be quite difficult as there is a lack of widely recognized and clearly defined resources that provide a comprehensive explanation. Although ...
- 11
0
votes
1
answer
73
views
Is there a recommended resource that can provide a detailed overview of the gradient norm?
When it comes to the concept of "Gradient Norm," it can be challenging to find a widely recognized and clearly defined resource that offers a comprehensive explanation. While many search ...
- 11
2
votes
1
answer
373
views
How is the max function differentiable wrt multiple arguments?
I recently came across an answer on StackOverflow that mentioned the max function being differentiable with respect to its values.
From my current understanding of ...
- 554
10
votes
2
answers
806
views
Why use ReLU over Leaky ReLU?
From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive.
With just this info to go off of,...
- 103
0
votes
0
answers
104
views
Computational overhead of "SCALING FORWARD GRADIENT WITH LOCAL LOSSES"
The paper "SCALING FORWARD GRADIENT WITH LOCAL LOSSES" discusses a new way of training deep neural networks called forward gradient learning. This method is different from the traditional ...
- 101
0
votes
0
answers
29
views
Is it possible to get recurrence through alternating forward and backward propagation?
This is a purely theoretical question, and I'm not trying to solve some real world problem with this, and I don't care about how ineffective this may be, as long as it is theoretically possible.
For a ...
- 101
0
votes
0
answers
48
views
Backpropagation of position-wise feedforward neural network
I have read a paper entitled "Attention is all you need" by Vaswani et al. (2017). This paper use the so-called position-wise feedforward neural network, where the input of this network is a ...
- 23
0
votes
0
answers
103
views
Why Hadamard product appear in General backpropagation derivation?
I'm trying to understand the basic backpropagation of a neural network and stuck with this formula, which describes how you calculate the error vector for a layer $i$:
$$\delta z^{[i]} = (w^{[i + 1]})^...
3
votes
1
answer
101
views
What is the partial derivative $\frac{\partial y}{\partial x_1}$ in this neural network?
The answer is supposed to be -6, but I don't know how to get that.
Also, in a NN, is that 2nd hidden layer possible, where the neurons are not dependent on all the neurons of the previous layer?
- 33
1
vote
1
answer
110
views
How do policy gradients work?
If I understand it correctly from the following equation
$$U(\theta)=\mathbb{E}_{\tau \sim P(\tau;\theta)}\left [ \sum_{t=0}^{H-1}R(s_t,u_t);\pi_{\theta} \right ]=\sum_{\tau}P(\tau;\theta)R(\tau)$$
...
- 175
0
votes
1
answer
43
views
How would I use backpropagation with a changing cost function [closed]
I have a neural network that is being trained with a changing cost function. Could I use backpropagation at all? If yes, how would I do this?
- 11
1
vote
2
answers
63
views
Why this single layer perceptron for the add operation not learning the correct weights?
This might be unnecessary but to learn the basics of neural networks, I am trying to create a single perceptron neural network to solve the adding operation of 2 inputs ...
- 193
0
votes
0
answers
124
views
Can an LLM such as GPT learn from a sentence prediction that is close in meaning but drastically different in wording?
Consider this example: husband cheats on wife, AI predicts that wife gets angry. AI predicts that wife may either scream or throw things, with screaming a higher probability. Turns out in reality ...
- 235
1
vote
1
answer
134
views
Is the re-parameterization trick necessary in the policy gradient method?
If we want to learn a stochastic policy with the policy gradient method, we have to sample from the distribution to get an action.
Wouldn't this lead to the same issue that variational autoencoders ...
- 175
0
votes
1
answer
66
views
Seq2seq with RNNs, how is the training loop performed?
How do we train a seq2seq rnn training?
We input a sentence that needs to be translated. We encode it sequentially. Then the first decoder outputs the first word with probabilities. We do a gradient ...
1
vote
3
answers
88
views
Do Artificial Neural Network with non-linear activation only in the output layer follows linearity?
I am using a model with linear activation in the hidden layer and non-linear activation in the output layer. Could you please help to know whether such models exhibit linearity or not?
The non-linear ...
0
votes
0
answers
10
views
Multiple network with one loss function, non-differentiable, is it okay with backpropagation?
Hello I'm trying to make the augmentation function which chooses to augmentate or not based on the contents of image.
To make this, I'm thinking about using two different network.
One is to classify ...
- 1
0
votes
0
answers
42
views
How does the cross entropy loss function interact with the final layer of a neural network?
I am having trouble understanding how the result of categorical cross entropy loss can be used to calculate the gradient for all of the weights.
The output of cross entropy function is the sum of all ...
- 1
2
votes
0
answers
62
views
Backpropagation with multiple output neurons but only one loss value
Suppose we have the following neural network (in reality it is a CNN with 60k parameters):
This image, as well as the terminology used here, is borrowed from Matt Mazur
As is visible, there are two ...
0
votes
0
answers
86
views
When should I think of using the Forward Forward algorithm?
Recently I read the paper named "The Forward-Forward Algorithm: Some Preliminary Investigations" and I was wondering what cases I should think of using it on.
So according to the paper If we ...
0
votes
1
answer
265
views
Single Layer Perceptron Backpropagation: How to compute affect of the net value on the output?
Assuming a single perceptron (see figure), I have found two versions of how to use backpropagation to update the weights. The perceptron is split in two, so we see the weighted sum on the left (the ...
- 1
0
votes
0
answers
41
views
Why differentiation of fourier operator is difficult?
I have a question when I read some papers about physics-informed neural networks.
In the paper of physics-informed neural operator, they said "it is non-trivial to compute the derivatives for ...
- 1
0
votes
1
answer
72
views
Is it possible to reverse engineer out the loss based on weights update when data is unknown?
Assume the gradient updates (both $W_t$ and $W_{t+1}$) and learning rate are known while data $X$ is unknown, is it possible to deduce the loss $L$ used in backprop algorithm that gave rise to the ...
- 175
1
vote
2
answers
1k
views
Why should one expect the backward pass to take twice as long as the forward pass?
I have seen it stated that, as a rule of thumb, a backward pass in a neural network should take about twice as long as the forward pass. Examples:
From DeepSpeed's Flops Profiler docs, the profiler:
...
- 111
1
vote
2
answers
101
views
Back propagation activation function derivative [closed]
I am reading about backpropagation for fully connected neural networks and I found a very interesting article by Jeremy Jordan. It explains the process from start to finish. There is a section though ...
1
vote
0
answers
26
views
How to make my neural networks designs more robust
Whenever, I design a neural network to solve a novel problem (requires a novel loss function i.e. not image classification) it always ends up being hypersensitive to batch size and learning rate.
...
- 326
0
votes
2
answers
233
views
How Does Convolution Backpropagation Work?
Assume in a convolutional layer's forward pass we have a $10\times10\times3$ image and five $3\times3\times3$ kernels, then $(10\times10\times3) *( 3\times3\times3\times5)$ has the output of ...
- 21
1
vote
0
answers
71
views
Training loss is decreasing very slowly while learning MNIST database
I am developing my ANN from scratch which is supposed to classify MNIST database of handwritten digits (0-9). My feed-forward fully connected ANN has to be composed of:
One input layer, with ...
- 11