Questions tagged [backpropagation]

For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.

Filter by
Sorted by
Tagged with
3
votes
1answer
137 views

Why is the backpropagation algorithm used to train the multilayer perceptron?

I've read in the book Neural Network Design, by Martin Hagan et al. (chapter 11), that, to train the feed-forward neural network (aka multilayer perceptron), one uses the backpropagation algorithm. ...
1
vote
0answers
83 views

How are weight matrices in attention learned?

I have been looking into transformers lately and have been reading tons of tutorials. All of them address the intuition behind attention, which I understand. However, they treat learning the weight ...
2
votes
0answers
46 views

How is the gradient with respect to weights derived in batch normalization?

At the bottom of page 2 of the paper L2 Regularization versus Batch and Weight Normalization, the equation for the gradient of the output with respect to the weights is given as: $$ \triangledown y_{...
4
votes
1answer
102 views

Backpropagation equation for a variant on the usual Linear Neuron architecture

Recently I encountered a variant on the normal linear neural layer architecture: Instead of $Z = XW + B$, we now have $Z = (X-A)W + B$. So we have a 'pre-bias' $A$ that affects the activation of the ...
0
votes
1answer
53 views

Why "large set of training data" is needed in Neural Network AI training?

I often heard people saying, "large set of training data is needed for producing an accurate AI". But when I looked for articles explaining backpropagations online, it all seems like you ...
3
votes
0answers
25 views

How are partial derivatives calculated in a computational graph?

I am trying to understand how are partial derivatives calculated in a computational graph. I understand reasoning behind computational graphs and I am bold enough to say I understand how they work, at ...
1
vote
1answer
40 views

Positive bias causes the calculation of incorrect gradients

I have a data set with a positive bias (an image, where the values range from 0 to 1), that seems to be causing my network to calculate incorrect gradients. If I just use the raw image as input, of ...
2
votes
0answers
24 views

CNN: Difficulties understanding backward pass derivatives

I have really quite hard difficulties to understand what is actually going on in the backward pass of a CNN. I am currently focusing on these references: https://towardsdatascience.com/forward-and-...
34
votes
4answers
32k views

What is the time complexity for training a neural network using back-propagation?

Suppose that a NN contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_i$ nodes in each layer. What is the time complexity to train this NN using back-propagation? I have a basic ...
3
votes
1answer
129 views

How to perform back-propagation in Decoupled Neural Interfaces?

I am attempting to create a fully decoupled feed-forward neural network by using decoupled neural interfaces (DNIs) as explained in the paper Decoupled Neural Interfaces using Synthetic Gradients (...
0
votes
0answers
61 views

What are the input and output gradients in PyTorch?

Suppose I want to train a neural network with $m-$length inputs of form $x = [x_1, x_2, x_3, \cdots, x_m]$ and $n-$length outputs of form $y = [y_1, y_2, y_3, \cdots, y_n]$. Let the number of ...
1
vote
0answers
23 views

In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?

Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one ...
2
votes
1answer
265 views

Training the generator in a GAN pair with back propagation

For the purposes of this question I am asking about training the generator, assume that training the discriminator is another topic. My understanding of generative adversarial networks is that you ...
1
vote
1answer
37 views

Does the ANN's training data include the proper output for every neuron?

I was designing an Artificial Neural Network a while back, but hit a bump when I got to the backpropagation. I was having trouble making the script choose whether to add or subtract from the weights, ...
2
votes
0answers
39 views

How does backpropagation know which weights to change?

I'm currently working on constructing a neural network from scratch (in JavaScript). I'm in the middle of working on the backpropagation, but there's something I don't understand: how does the ...
1
vote
1answer
53 views

What is the correct formula for updating the weights in a 1-single hidden layer neural network?

I'm creating a neural network with 3 layers and no bias. On internet I saw that the expression for the derivative of the weights between the hidden layer and the output layer was: $$\Delta W_{j,k} = (...
1
vote
1answer
25 views

Parallelize Backpropagation - How to synchronize the weights of each thread?

I implemented a parallel backpropagation algorithm that uses $n$ threads. Now every thread gets $\dfrac{1}{n}$ examples of the training data and updates its instance of the net with it. After every ...
3
votes
1answer
117 views

What is asymmetric relaxation backpropagation?

In Chapter 8, section 8.5.2, Raul Rojas describes how the weights for a layer of a neural network can be calculated using a pseudoinverse of the sigmoid function in the nodes, he explains this is an ...
2
votes
2answers
150 views

Which neuron represents which part of the input?

In a neural network, each neuron represents some part of the input. For example, in the case of a MNIST digit, consider the stem of the number 9. Each neuron in the NN represents some part of this ...
0
votes
0answers
18 views

How Long Can BPTT Truncated?

I wanted to ask what is, in general, the maximum value (the order of magnitude) of the number of time steps I can back-propagate in the past using TBTT (Truncated Backpropagation Through Time) in an ...
2
votes
1answer
89 views

Computation of initial adjoint for NODE

I'm reading the paper Neural Ordinary Differential Equations and I have a simple question about adjoint method. When we train NODE, it uses a blackbox ODESolver to compute gradients through model ...
1
vote
1answer
63 views

Different ways to calculate backpropagation derivatives, any difference?

I'm studying error backpropagation in neural networks. I am interested in why we use only one path on the computational graph to get the value of the derivative for a weight? I ask the question ...
1
vote
0answers
18 views

Discrepancy of backpropagation formula between Andrew Ngs ML Course and those derived by neuralnetworksanddeeplearning.com

I'm currently working through Week 5 of Andrew Ngs Machine Learning course on Coursera, which goes through the backprop algorithm for basic neural networks. Whilst trying to derive the formulae he ...
0
votes
0answers
18 views

RNN - Backpropagation through time - Gradient Calculation

I think I got it right after reading multiple resources but im still not 100%. Seems like everyone is calculating it different. Or they just shortcut explaining the calculation. (or my math skills ...
3
votes
0answers
45 views

How does back propagation adjust the hidden layers' weights and biases?

I'm new to neural networks and trying to figure out its fundamentals but I cannot fully understand the back propagation algorithm. In back propagation, I understand we want to go backwards from the ...
2
votes
3answers
227 views

Backpropagation of neural nets with shared weight

I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. ...
1
vote
1answer
71 views

How does a single neuron in hidden layer affect training accuracy [duplicate]

I'm currently a student learning about AI Networks. I've came across a statement in one of my Professor's books that a FFBP (Feed-Forward Back-Propagation) Neural Network with a single hidden layer ...
3
votes
1answer
48 views

What does "differentiable architecture" mean?

I'm currently reading a paper that uses CNN's as a base approach to solving some image classification issues and I've found that they kept mentioning the term "Differentiable Architecture", ...
1
vote
0answers
23 views

ReLU function converging to local optimum in one case and diverging in the other one

I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function. To check my implementation I ...
2
votes
1answer
77 views

How to create a neural network from a set of equations?

Say I have these equations: $$x_1 = x_2 + 2y_1 + b$$ $$x_2 = y_2 + c$$ $$y_1 = z + a$$ $$y_2 = y_3 + d$$ $$z = z_1 + e$$ $x_1$ depends on $x_2$ (depends on $y_2$ (depends on $y_3$)) and $y_1$ (depends ...
1
vote
0answers
46 views

Why doesn't anyone use reinforcement learning to find the best possible alternative to backpropagation?

To be clear, I'm very uninformed on the topic of alternative learning algorithms to backprop, all my knowledge comes from articles like these: lets-not-stop-at-backprop backprop-alternatives we-need-a-...
0
votes
0answers
13 views

when updating the bias matrix, do we get the total sum of dZ or the sum of the axis of dZ?

I'm currently studying how to implement a neural network from scratch to know how it works, I came across this article: https://www.samsonzhang.com/2020/11/24/understanding-the-math-behind-neural-...
1
vote
0answers
32 views

Does Keras or Tensorflow calculate backpropagation?

I am using tensorflow 2.1.0. Does Keras or Tensorflow calculate backpropagation? Where can I find the code for backpropagation in Keras and Tensorflow? Thanks, Jianqiao Huang
10
votes
5answers
625 views

What is "backprop"?

What does "backprop" mean? Is the "backprop" term basically the same as "backpropagation" or does it have a different meaning?
3
votes
1answer
58 views

Can some of the weights be fixed during the training of a neural network?

Is it possible to exclude specific layers from the optimization? For example, let's say I have an input layer, 2 hidden layers, and the output layer. I know there is a perfect solution for my problem ...
1
vote
0answers
46 views

Backpropagation not working as expected

I'm new to neural networks and I try to make a model that is guessing if a point is below or above relative to a function output. The idea is inspired from this video https://youtu.be/DGxIcDjPzac . ...
0
votes
0answers
19 views

Gradient of Scalar objective cannot be efficiently calculated?

Suppose we generate the vector output $y$ from model $h(x, \theta)$, with input $x$ and parameters $\theta$. Reverse mode differentiation says that we can calculate the gradient \begin{align*} \...
0
votes
0answers
21 views

Are there any good references that describe the equations of the forward pass of Graph Neural Networks?

I am trying to program Graph Neural Network from scratch. Can the community please suggest a good reference/s to read about the equations of the forward pass in Graph Neural Networks, especially in ...
1
vote
1answer
179 views

What is the difference between batches in deep Q learning and supervised learning?

How is the batch loss calculated in both DQNs and simple classifiers? From what I understood, in a classifier, a common method is that you sample a mini-batch, calculate the loss for every example, ...
1
vote
0answers
153 views

Convolutional Layer Multichannel Backpropagation Implementation

I have been working on coding a CNN in python from scratch using numpy as a semester project and I think I have successfully implemented it up to backpropagation in the MaxPool Layers. However, my ...
0
votes
0answers
24 views

Do $V_\theta$ and $V_s$ represent partial or total derivatives in the paper "Learning Continuous Control Policies by Stochastic Value Gradients"?

I was reading up on the Stochastic Value Gradients paper by Heess et al. In the paper, they describe a recursive process to calculate path-wise derivatives via equations (3) and (4), at the bottom of ...
1
vote
0answers
24 views

Backpropagation - what does rate of change calculated from the partial derivatives actually relate to?

I understand conceptually how backpropagation works according to the chain rule, and I understand that partial derivatives calculate the rate of change of a function containing multiple variables with ...
1
vote
1answer
295 views

Simple three layer neural network with backpropagation is not approximating tanh function

I have this simple neural network in Python which I'm trying to use to aproximation tanh function. As inputs I have x - inputs to the function, and as outputs I want tanh(x) = y. I'm using sigmoid ...
4
votes
1answer
122 views

Is the bias also a "weight" in a neural network?

I'm learning about how neural networks are trained. I understand how a neuron works, backpropagation, and all that. In neurons, there is a clear distinction between a "weight" and a "...
0
votes
0answers
42 views

Double DQN backpropagation of negative final rewards?

My problem is that in my Double DQN model, negative final rewards are not being backpropagated into action Q-values, and so some Q-values are positive, when they should be negative, and hence ...
0
votes
0answers
53 views

Backpropagation in REINFORCE algorithms with Categorical / Multinomial Distribution

From a paper by Williams, I know in general how to backpropagate log-probabilities of chosen actions when applying the REINFORCE weight update rule. However, I was wondering about a case not being ...
3
votes
1answer
48 views

How to improve a trained model over time (i.e. with more predictions)?

I built a model using the tutorial on the TensorFlow site. It was a simple image classification neural network. I trained it and saved the model and weights together on a ...
2
votes
0answers
18 views

In a convolutional neural network, how is the error delta propagated between convolutional layers?

I'm coding some stuff for CNNs, just relying on numpy (and scipy just for the convolution operation for pure performance reasons). I've coded a small network consisting of a convolutional layer with ...
2
votes
1answer
154 views

Why is it a problem if the outputs of an activation function are not zero-centered?

In this lecture, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that ...
2
votes
1answer
84 views

What are examples of good free books that cover the back-propagation algorithm?

What are examples of good free books that cover the back-propagation used to train multilayer perceptrons? I've just started to learn about artificial neural networks, so I'm looking for books that ...

1
2 3 4 5