Questions tagged [backpropagation]

For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.

216 questions
Filter by
Sorted by
Tagged with
49 views

Convolutional Layer Multichannel Backpropagation Implementation

I have been working on coding a CNN in python from scratch using numpy as a semester project and I think I have successfully implemented it up to backpropagation in the MaxPool Layers. However, my ...
33 views

49 views

Why is the error curve of a neural network trained with MSE to output $\frac{3 I_1 + 5 I_2}{2}$ given inputs $I_1$ and $I_2$ oscillating weirdly?

I just "finished" my first AI program. I programmed in Excel VBA, and I think it works well. I was checking every formula and the whole algorithm several times to make sure every formula is ...
29 views

Positive bias causes the calculation of incorrect gradients

I have a data set with a positive bias (an image, where the values range from 0 to 1), that seems to be causing my network to calculate incorrect gradients. If I just use the raw image as input, of ...
119 views

How to perform back-propagation in Decoupled Neural Interfaces?

I am attempting to create a fully decoupled feed-forward neural network by using decoupled neural interfaces (DNIs) as explained in the paper Decoupled Neural Interfaces using Synthetic Gradients (...
70 views

Bias gradient of layer before batch normalization always zero

From the original paper and this post we have that batch normalization backpropagation can be formulated as I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
48 views

Why is Openai's PPO2 implementation differentiable?

I'm trying to understand the concept behind the implementation of the OpenAI PPO2 algorithm. The loss function that is minimized is as follows: ...
215 views

Training the generator in a GAN pair with back propagation

For the purposes of this question I am asking about training the generator, assume that training the discriminator is another topic. My understanding of generative adversarial networks is that you ...
71 views

Why is tf.abs non-differentiable in Tensorflow?

I understand why tf.abs is non-differentiable in principle (discontinuity at 0) but the same applies to tf.nn.relu yet, in case of this function gradient is simply set to 0 at 0. Why the same logic is ...
26 views

Are all activation functions in a layer same? [duplicate]

I understand that for you can have multiple activation functions in different layers. CNN's usually have Relu followed by softmax for the classification. But what stops us in having multiple ...
23 views

Why are the weights of the previous layers updated only considering the old values of the weights of the later layer, not the updated values?

Why are the weights of a neural net updated only considering the old values of the later layer, not the already updated values? I use this example to explain my problem. When applying the ...
41 views

Why does sigmoid saturation prevent signal flow through the neuron?

As per these slides on page 35: Sigmoids saturate and kill gradients. when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero. the gradient and ...
2k views

What is the best XOR neural network configuration out there in terms of low error?

I'm trying to understand what would be the best neural network for implementing an XOR gate. I'm considering a neural network to be good if it can produce all the expected outcomes with the lowest ...
98 views

What is asymmetric relaxation backpropagation?

In Chapter 8, section 8.5.2, Raul Rojas describes how the weights for a layer of a neural network can be calculated using a pseudoinverse of the sigmoid function in the nodes, he explains this is an ...
697 views

In deep learning, is it possible to use discontinuous activation functions?

In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)? (My guess: for example, ReLU is non-differentiable at a single point, but it still has a ...
121 views

Which neuron represents which part of the input?

In a neural network, each neuron represents some part of the input. For example, in the case of a MNIST digit, consider the stem of the number 9. Each neuron in the NN represents some part of this ...
589 views

What are the learning limitations of neural networks trained with backpropagation?

In 1969, Seymour Papert and Marvin Minsky showed that Perceptrons could not learn the XOR function. This was solved by the backpropagation network with at least one hidden layer. This type of network ...
68 views

How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?

I've seen plenty of examples of people doing Sigmoid + MSE backpropagation implementations, yet I do not seem to understand how to implement backpropagation as stated in the title in the case of multi-...
182 views

How does the neural-network know how to tweak weights for a specific neuron?

I know backpropagation uses cost and gradient descent to tweak the weights to minimize the cost. But how does it know which weights to give more weight to in the first place? Is there something inside ...
1k views

Do we know what the units of neural networks will do before we train them?

I was learning about back-propagation and, looking at the algorithm, there is no particular 'partiality' given to any unit. What I mean by partiality there is that you have no particular ...
115 views

Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?

I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation. As per my knowledge, ...
18 views

Neural Network trains towards 1 despite target

So I'm trying to make my first neural network and have just finished my back propagation functions. I got the algebra from brilliant and thought I'd understood it, but my bug proves otherwise. The bug ...
23 views

During Backpropagation in LSTM, why is the previous output $h_{t-1}$ considered constant w.r.t any $W$ while computing derivative?

I've just started learning LSTM, and some points in the process of calculating the gradients are getting me confused. Say, for example, we want to compute $\frac{\partial}{\partial W_i}L$, where $L$ ...
27 views

How do gradients are flown back into the Siamese network when branching is done?

I am curious about the working of a Siamese network. So, let us suppose I am using a triplet loss for my network and I have instantiated single CNN 3 times and there are 3 inputs to the network. So, ...
30 views

Backpropagation implementation not applicable for other cases

I saw this implementation of backpropagation in MATLAB, where the loss function used is MSE, and the last layer's activation function was sigmoid. I denoted the portions of the formula for what I ...
17 views

For the generalised delta rule in back-propogation, do you subtract the target from the obtained output, or vice versa?

When I look up the generalised delta rule equation for back-propogation, I am seeing two conflicting equations. For example, here (slide 20), given $o$ (the output, defined in slide 18), $z$ (the ...
I understand that this is the method for conducting back-propogation: With a three layer network (input layer, one hidden layer, one output layer); start with input $I_i$ as an exemplar input. ...