# Questions tagged [backpropagation]

For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.

286 questions
Filter by
Sorted by
Tagged with
14 views

### If weights and biases are in superposition state, do we still need backpropagation algorithm?

Deep learning is basically finding the best weights and biases in order to reduce loss that is computed by the defined loss function to be as small as possible (global minimum). Finding the global ...
14 views

### Backpropagation with rasterization step

I have an odd little problem facing me for my project. I have a smooth polygon defined by parameters. I have convolution transformation, similar to a Gaussian blur. This transformation can only be ...
• 11
28 views

### Exact backpropagation formula for gradients vs online source

I visited this website that showed a simple backpropagation technique being applied on a built-from-scratch ANN. The code is in python and looks like this: ...
1 vote
347 views

### How does BackProp avoid bias for the last input used for training?

Here's a BackProp Algo definition from here: Initially all the edge weights are randomly assigned. For every input in the training dataset, the ANN is activated and its output is observed. This ...
33 views

### Backpropagation math question

Hi I have a very simple model and I'm trying to learn the math of it. Basically, I have an input matrix X m x n. An output matrix Y m x n is formed from some convolution H. The figure of merit is ...
• 11
26 views

### Not Averaging Gamma and Beta Gradients in BatchNormalization leads me to higher accuracy

I'm implementing batchnorm from scratch in pure NumPy. I noticed something interesting. While I'm calculating the gradients of gamma (dg) and beta (db), ignoring the summation / averaging of the ...
• 125
101 views

### Back-propagation through embedding layer

I was trying to understand how back-propogation works through embedding layer. After I wrote the equation, it seems like the gradient with respect to the weight of embedding layer is always 1.  \...
39 views

### Creating a replacement for backpropagation through evolution [closed]

Can we create a learning algorithm that solves all the problems of backpropagation through evolution?
26 views

### Neural Network with Incorrect Calculation Better than Correct One

I have designed my own neural network and discovered an error. During backpropagation, instead of inserting the Z-values into the derivative of the activation function, I inserted the A-values. The ...
506 views

### Is it easier to use back-propagation or genetic algorithms to teach an artificial intelligence?

I am making a very simple neural network for a school project, and I would like to know what the best and easiest way to "teach" a neural network would be. From what I know, backpropagation ...
37 views

### Is my C# Adam implementation correct?

I have some doubt because I incurred in different papers proposing different implementations. Also implementations on opensource projects looks different. In example there is a C++ library that ...
86 views

### Total loss in backpropagation

I'd say I have some understanding of backpropagation, however I am not really sure of the total loss being calculated. Let us take the example below: After 1 forward pass when I have to update the ...
47 views

### How to optimise a FNN/MLP network with MSE (positive only loss) in C

I can create a FNN/MLP network in C but only g-p loss works, where g = ground truth and p = predicted. What I don't understand is how MSE a positive only loss value can train a back propagation ...
1 vote
73 views

### Why do we update parameters with respect to cost instead of loss? [closed]

Assuming that the cost function $J$ is the average of the loss function $\mathcal{L}$ over all training examples $m$ $J(w) = \frac{1}{m} \displaystyle\sum_{i=0}^{m} \mathcal{L}$ why do we update the ...
• 121
62 views

### Reinforcement learning - calculating policy gradient using cross entropy loss

I am writing a program that uses reinforcement learning and the policy gradient method to play Pong. It basically extends Andrej Karpathy's version (https://gist.github.com/karpathy/...
60 views

### Why does "in-place" mutation cause automatic differentiation to fail, and how to write code to avoid this problem

I work with a few different automatic differentiation frameworks, including pytorch, Jax, and Flux in Julia. Periodically I run some code and I get errors about mutations or operations occurring "...
• 207
279 views

### What do you mean by "updating based on a training example/batch" in Gradient Descent?

My understanding is this: When doing Stochastic Gradient Descent over a neural network, in every epoch, we run $n$ iterations (where the dataset has $n$ training examples) and in every iteration, we ...
64 views

### Can you explain the Hinton's comment "Rprop is equivalent to using the gradient, but also dividing by the size of the gradient"?

Been reviewing some old foundational material and ran into this comment by Hinton on Rprop in his old Coursera class: Rprop is equivalent to using the gradient, but also dividing by the size of the ...
• 121
11 views

### Neural Net Convergence for Batch SGD

I've built a dynamically sized neural network framework with for multi class classification—just to strengthen my understanding of the deep networks. I'm training and predicting my network to classify ...
37 views

### Can positional encodings in transformers be added

Here's a basic GPT2 implementation: ...
• 153
45 views

### How to prevent update a pretrained model if a model is optimized with backpropagation? [closed]

These are components in my model: A generator An encoder: a pretrained, and should not updated. A loss function. Input is passed to the encoder to generate X, X is then passed to generator to ...
• 109
40 views

### What am I doing wrong that result in a graph indicating better gradients in non-scaled dot-product attention compared to the scaled version?

I'm trying to visualize how the gradients change as we're increasing $d_{k}$ in the scaled dot-product attention and compare it to its non scaled version but I'm failing to produce a reasonable graph ...
• 575
1 vote
83 views

### In multilayer perceptron, to update the Bias value, do we use the neuron's delta, or do we use the sum of all deltas?

In multilayer perceptron, to update the Bias value, do we use the neuron's delta, or do we use the sum of all deltas? I ask this because I once saw a video on YouTube where the formula for updating ...
• 267
75 views

### In MLP, in the formula to calculate the delta in the hidden layer, it uses the sum of all Deltas and Connection Weights in the next layer?

Reference: http://neuralnetworksanddeeplearning.com/chap2.html I have a question about the equation to calculate the delta in the hidden layer: In the formula to calculate the delta in the hidden ...
• 267
20 views

### In MLP, Why in the formula to calculate delta, the derivation of the cost function "(predicted - expected)" just as "(predicted - expected)" itself?

I know that we have to perform the derivation of the terms, using the chain rule, to obtain the equations to calculate the gradients of the neurons. In this article: https://machinelearningmastery.com/...
• 267
1 vote
40 views

### In MLP, to calculate the delta, do I need to calculate the derivative of the cost function? Or can I just use the cost function result?

In Multi Layer Perceptron networks, I have a question: in the formula to calculate the error in the output layer, Some articles say it is like this ...
• 267
206 views

### How is back propagation applied in case the activation function is not differentiable?

Back propagation is based on partial derivatives including the activation functions. How is back propagation applied when the activation function is not differentiable?
• 974
40 views

### NaN gradients while Training (but loss isn't NaN neither the computational graph is disconnected)

The reason I am asking this here is that I haven't found a bug in my code and maybe there isn't a bug at all (or maybe there is). I just want to validate the idea that I am trying to implement. Here ...
• 101
1 vote
115 views

### Why in one article to calcule the delta uses the weights of current layer, while in another article uses the weights of neuron inputs of next layer?

Article 1: https://pyimagesearch.com/2021/05/06/backpropagation-from-scratch-with-python/ Article 2: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ I was ...
• 267
1 vote
38 views

### Neural network learns to mimic distribution of classes in dataset instead of using signal from input

I'm trying to implement example from a classic AI paper named "Learning representations by back-propagating errors" by Hinton et al. Example aims at training network able to predict third ...
34 views

### How was the word2vec model trained?

Let's take the CBOW (continuous bag of words) model as the example. Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
45 views

### If I manually trained a multilayer percetron neural network, always following exactly the same steps meticulously, would I always get the same result?

Reference: https://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html If I trained a multilayer percetron neural network manually, following exactly the backpropagation steps described in the article, ...
• 267
126 views

### Does this article make use of the chain rule? And where?

References: Chain rule in Wikipedia: https://en.wikipedia.org/wiki/Chain_rule Chain rule in Towards Data Science: https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd#:~...
• 267
1 vote
63 views

### In multilayer perceptron neural networks, are the names "delta", "gradient" and "error" all the same thing? or not?

What is the difference between "delta", "gradient" and "error", are these names the same thing? I'm confused because someone once told me that both the names "delta&...
• 267
1 vote
91 views

### Please, could someone help me understand if the backpropagation explanations in these two articles about calculate the error are equivalent?

I have a question about backpropagation, I'm a beginner, I'm studying the formulas to calculate the delta of neurons, there are several sources on the internet, which teach in different ways, so I'm ...
• 267
1 vote
475 views

### what is the backpropagation formula to calculate delta and update weights?

I'm trying to study how backpropagation works step by step in a MultiLayer Perceptron neural network. I would really like to be able to understand how these calculations work. And I have a specific ...
• 267
34 views

### Independent parameter update in backpropagation

When we calculate the gradient wrt to each paramters, we consider the other parameters remain constant, but the moment their is a change in any of the other parameters, shouldn't all the other changes ...
115 views

• 121
1 vote
406 views

• 23
69 views

### Is there a resource that offers a detailed overview of the gradient flow?

Understanding the concept of "Gradient Flow" can be quite difficult as there is a lack of widely recognized and clearly defined resources that provide a comprehensive explanation. Although ...
146 views

### Is there a recommended resource that can provide a detailed overview of the gradient norm?

When it comes to the concept of "Gradient Norm," it can be challenging to find a widely recognized and clearly defined resource that offers a comprehensive explanation. While many search ...
• 11
647 views

### How is the max function differentiable wrt multiple arguments?

I recently came across an answer on StackOverflow that mentioned the max function being differentiable with respect to its values. From my current understanding of ...
• 584