Questions tagged [backpropagation]
For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.
286
questions
0
votes
0
answers
14
views
If weights and biases are in superposition state, do we still need backpropagation algorithm?
Deep learning is basically finding the best weights and biases in order to reduce loss that is computed by the defined loss function to be as small as possible (global minimum).
Finding the global ...
0
votes
0
answers
14
views
Backpropagation with rasterization step
I have an odd little problem facing me for my project.
I have a smooth polygon defined by parameters.
I have convolution transformation, similar to a Gaussian blur. This transformation can only be ...
- 11
0
votes
0
answers
28
views
Exact backpropagation formula for gradients vs online source
I visited this website that showed a simple backpropagation technique being applied on a built-from-scratch ANN. The code is in python and looks like this:
...
1
vote
1
answer
347
views
How does BackProp avoid bias for the last input used for training?
Here's a BackProp Algo definition from here:
Initially all the edge weights are randomly assigned. For every input in the training dataset, the ANN is activated and its output is observed. This ...
0
votes
0
answers
33
views
Backpropagation math question
Hi I have a very simple model and I'm trying to learn the math of it.
Basically, I have an input matrix X m x n. An output matrix Y m x n is formed from some convolution H. The figure of merit is ...
- 11
0
votes
0
answers
26
views
Not Averaging Gamma and Beta Gradients in BatchNormalization leads me to higher accuracy
I'm implementing batchnorm from scratch in pure NumPy.
I noticed something interesting.
While I'm calculating the gradients of gamma (dg) and beta (db), ignoring the summation / averaging of the ...
- 125
0
votes
0
answers
101
views
Back-propagation through embedding layer
I was trying to understand how back-propogation works through embedding layer. After I wrote the equation, it seems like the gradient with respect to the weight of embedding layer is always 1.
$$
\...
-1
votes
1
answer
39
views
Creating a replacement for backpropagation through evolution [closed]
Can we create a learning algorithm that solves all the problems of backpropagation through evolution?
0
votes
0
answers
26
views
Neural Network with Incorrect Calculation Better than Correct One
I have designed my own neural network and discovered an error. During backpropagation, instead of inserting the Z-values into the derivative of the activation function, I inserted the A-values. The ...
- 1
2
votes
2
answers
506
views
Is it easier to use back-propagation or genetic algorithms to teach an artificial intelligence?
I am making a very simple neural network for a school project, and I would like to know what the best and easiest way to "teach" a neural network would be. From what I know, backpropagation ...
- 23
0
votes
0
answers
37
views
Is my C# Adam implementation correct?
I have some doubt because I incurred in different papers proposing different implementations. Also implementations on opensource projects looks different.
In example there is a C++ library that ...
- 301
0
votes
1
answer
86
views
Total loss in backpropagation
I'd say I have some understanding of backpropagation, however I am not really sure of the total loss being calculated.
Let us take the example below:
After 1 forward pass when I have to update the ...
- 9
0
votes
1
answer
47
views
How to optimise a FNN/MLP network with MSE (positive only loss) in C
I can create a FNN/MLP network in C but only g-p loss works, where g = ground truth and p = predicted.
What I don't understand is how MSE a positive only loss value can train a back propagation ...
user81637
1
vote
2
answers
73
views
Why do we update parameters with respect to cost instead of loss? [closed]
Assuming that the cost function $J$ is the average of the loss function $\mathcal{L}$ over all training examples $m$
$J(w) = \frac{1}{m} \displaystyle\sum_{i=0}^{m} \mathcal{L}$
why do we update the ...
- 121
0
votes
0
answers
62
views
Reinforcement learning - calculating policy gradient using cross entropy loss
I am writing a program that uses reinforcement learning and the policy gradient method to play Pong. It basically extends Andrej Karpathy's version (https://gist.github.com/karpathy/...
- 1
0
votes
0
answers
60
views
Why does "in-place" mutation cause automatic differentiation to fail, and how to write code to avoid this problem
I work with a few different automatic differentiation frameworks, including pytorch, Jax, and Flux in Julia. Periodically I run some code and I get errors about mutations or operations occurring "...
- 207
3
votes
1
answer
279
views
What do you mean by "updating based on a training example/batch" in Gradient Descent?
My understanding is this: When doing Stochastic Gradient Descent over a neural network, in every epoch, we run $n$ iterations (where the dataset has $n$ training examples) and in every iteration, we ...
2
votes
1
answer
64
views
Can you explain the Hinton's comment "Rprop is equivalent to using the gradient, but also dividing by the size of the gradient"?
Been reviewing some old foundational material and ran into this comment by Hinton on Rprop in his old Coursera class:
Rprop is equivalent to using the gradient, but also dividing by the
size of the ...
- 121
0
votes
0
answers
11
views
Neural Net Convergence for Batch SGD
I've built a dynamically sized neural network framework with for multi class classification—just to strengthen my understanding of the deep networks. I'm training and predicting my network to classify ...
- 1
0
votes
1
answer
37
views
0
votes
1
answer
45
views
How to prevent update a pretrained model if a model is optimized with backpropagation? [closed]
These are components in my model:
A generator
An encoder: a pretrained, and should not updated.
A loss function.
Input is passed to the encoder to generate X, X is then passed to generator to ...
- 109
0
votes
0
answers
40
views
What am I doing wrong that result in a graph indicating better gradients in non-scaled dot-product attention compared to the scaled version?
I'm trying to visualize how the gradients change as we're increasing $d_{k}$ in the scaled dot-product attention and compare it to its non scaled version but I'm failing to produce a reasonable graph ...
- 575
1
vote
1
answer
83
views
In multilayer perceptron, to update the Bias value, do we use the neuron's delta, or do we use the sum of all deltas?
In multilayer perceptron, to update the Bias value, do we use the neuron's delta, or do we use the sum of all deltas?
I ask this because I once saw a video on YouTube where the formula for updating ...
- 267
0
votes
0
answers
75
views
In MLP, in the formula to calculate the delta in the hidden layer, it uses the sum of all Deltas and Connection Weights in the next layer?
Reference:
http://neuralnetworksanddeeplearning.com/chap2.html
I have a question about the equation to calculate the delta in the hidden layer: In the formula to calculate the delta in the hidden ...
- 267
0
votes
0
answers
20
views
In MLP, Why in the formula to calculate delta, the derivation of the cost function "(predicted - expected)" just as "(predicted - expected)" itself?
I know that we have to perform the derivation of the terms, using the chain rule, to obtain the equations to calculate the gradients of the neurons.
In this article: https://machinelearningmastery.com/...
- 267
1
vote
0
answers
40
views
In MLP, to calculate the delta, do I need to calculate the derivative of the cost function? Or can I just use the cost function result?
In Multi Layer Perceptron networks, I have a question: in the formula to calculate the error in the output layer, Some articles say it is like this ...
- 267
0
votes
1
answer
206
views
How is back propagation applied in case the activation function is not differentiable?
Back propagation is based on partial derivatives including the activation functions.
How is back propagation applied when the activation function is not differentiable?
- 974
0
votes
0
answers
40
views
NaN gradients while Training (but loss isn't NaN neither the computational graph is disconnected)
The reason I am asking this here is that I haven't found a bug in my code and maybe there isn't a bug at all (or maybe there is). I just want to validate the idea that I am trying to implement. Here ...
- 101
1
vote
1
answer
115
views
Why in one article to calcule the delta uses the weights of current layer, while in another article uses the weights of neuron inputs of next layer?
Article 1:
https://pyimagesearch.com/2021/05/06/backpropagation-from-scratch-with-python/
Article 2:
https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/
I was ...
- 267
1
vote
0
answers
38
views
Neural network learns to mimic distribution of classes in dataset instead of using signal from input
I'm trying to implement example from a classic AI paper named "Learning representations by back-propagating errors" by Hinton et al.
Example aims at training network able to predict third ...
- 11
0
votes
0
answers
34
views
How was the word2vec model trained?
Let's take the CBOW (continuous bag of words) model as the example.
Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
- 1
0
votes
1
answer
45
views
If I manually trained a multilayer percetron neural network, always following exactly the same steps meticulously, would I always get the same result?
Reference:
https://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html
If I trained a multilayer percetron neural network manually, following exactly the backpropagation steps described in the article, ...
- 267
0
votes
2
answers
126
views
Does this article make use of the chain rule? And where?
References:
Chain rule in Wikipedia: https://en.wikipedia.org/wiki/Chain_rule
Chain rule in Towards Data Science: https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd#:~...
- 267
1
vote
1
answer
63
views
In multilayer perceptron neural networks, are the names "delta", "gradient" and "error" all the same thing? or not?
What is the difference between "delta", "gradient" and "error",
are these names the same thing?
I'm confused because someone once told me that both the names "delta&...
- 267
1
vote
1
answer
91
views
Please, could someone help me understand if the backpropagation explanations in these two articles about calculate the error are equivalent?
I have a question about backpropagation, I'm a beginner, I'm studying the formulas to calculate the delta of neurons, there are several sources on the internet, which teach in different ways, so I'm ...
- 267
1
vote
1
answer
475
views
what is the backpropagation formula to calculate delta and update weights?
I'm trying to study how backpropagation works step by step in a MultiLayer Perceptron neural network. I would really like to be able to understand how these calculations work. And I have a specific ...
- 267
0
votes
0
answers
34
views
Independent parameter update in backpropagation
When we calculate the gradient wrt to each paramters, we consider the other parameters remain constant, but the moment their is a change in any of the other parameters, shouldn't all the other changes ...
0
votes
1
answer
115
views
Confusion over taking gradients in Variational Autoencoders (VAE)
I am confused as to when to hold certain parameters constant in a VAE. I will explain with a concrete example.
We can write $\operatorname{ELBO}(\phi, \theta) = \mathbb{E}_{q_{\phi}(z)}\left[\log
\...
0
votes
0
answers
43
views
neural network not training correctly
i'm trying to learn how a neural network works. i'm writing the neural network in C for handwritten digit recognition and training it on MNIST dataset. the neural network has an input layer with 28*28 ...
- 9
0
votes
1
answer
466
views
How to do backpropagation with argmax?
I am attempting to utilize two networks: a classifier and a linear network. Based on the output class of the first network, my goal is to retrieve the corresponding value from the linear network using ...
0
votes
1
answer
176
views
Does the output of LLM's affect their neural weights?
When an LLM creates an output, it seemingly has no way to check if its output was valid. Therefore it wouldn't be able to back-propagate any changes to the weights is used to create that output.
...
- 103
1
vote
2
answers
185
views
Backpropagation with CrossEntropy and Softmax, HOW?
Let Zs be the input of the output layer (for example, Z1 is the input of the first neuron in the output layer), Os be the output of the output layer (which are actually the results of applying the ...
- 11
1
vote
0
answers
48
views
Can gradient descent cause loss to increase in some situations?
Is a gradient descent step always supposed to decrease loss? I can think of a situation where it would seem that gradient descent would increase loss but maybe it I am misunderstanding a part of ...
- 11
1
vote
2
answers
62
views
How to apply backpropagation when one layer of the network is a call-only function (no gradient)?
I worked with Feed Forward Neural Network and VAE and understood backpropagation algorithm. Now I build a VAE network, one layer of it is a very complex vector-to-vector function $f(x)$ (a general '...
- 111
0
votes
1
answer
64
views
About the requirement to compute the gradient at layer $l$
I'm trying to understand a line of my note.
Let's say there is a simple feedforward neural network that has $N$ layers, and for a given layer $l$, it has weight $W^l$, and $g^l$ is the gradient to ...
- 121
1
vote
1
answer
406
views
Is there any reference about backpropagation of the Transformer's multi-head layer?
Is there any reference about backpropagation of the Transformer's multi-head layer or multi-head attention (MHA)? I have searched various journals but have not found one yet.
- 23
2
votes
2
answers
69
views
Is there a resource that offers a detailed overview of the gradient flow?
Understanding the concept of "Gradient Flow" can be quite difficult as there is a lack of widely recognized and clearly defined resources that provide a comprehensive explanation. Although ...
- 21
0
votes
1
answer
146
views
Is there a recommended resource that can provide a detailed overview of the gradient norm?
When it comes to the concept of "Gradient Norm," it can be challenging to find a widely recognized and clearly defined resource that offers a comprehensive explanation. While many search ...
- 11
3
votes
1
answer
647
views
How is the max function differentiable wrt multiple arguments?
I recently came across an answer on StackOverflow that mentioned the max function being differentiable with respect to its values.
From my current understanding of ...
- 584
11
votes
1
answer
6k
views
Why use ReLU over Leaky ReLU?
From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive.
With just this info to go off of,...
- 113