Questions tagged [backpropagation]

For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.

77 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
0answers
54 views

Why does sigmoid saturation prevent signal flow through the neuron?

As per these slides on page 35: Sigmoids saturate and kill gradients. when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero. the gradient and ...
4
votes
0answers
33 views

How do weights changes handles during back-propagation when there are unknown labels

I have a question about how weights are updated during back-propagation for some of my samples that have unknown labels (please note, unknown, not missing). The reason they are unknown is because this ...
4
votes
1answer
429 views

How does backpropagation work on a custom loss function whose components have magnitudes of different orders?

I want to use a custom loss function which is a weighted combination of l1 and DSSIM losses. The DSSIM loss is limited between 0 and 0.5 where as the l1 loss can be orders of magnitude greater and is ...
3
votes
0answers
33 views

How are partial derivatives calculated in a computational graph?

I am trying to understand how are partial derivatives calculated in a computational graph. I understand reasoning behind computational graphs and I am bold enough to say I understand how they work, at ...
3
votes
1answer
122 views

What is asymmetric relaxation backpropagation?

In Chapter 8, section 8.5.2, Raul Rojas describes how the weights for a layer of a neural network can be calculated using a pseudoinverse of the sigmoid function in the nodes, he explains this is an ...
3
votes
0answers
157 views

What is the time complexity for training a gated recurrent unit (GRU) neural network using back-propagation through time?

Let us assume we have a GRU network containing $H$ layers to process a training dataset with $K$ tuples, $I$ features, and $H_i$ nodes in each layer. I have a pretty basic idea how the complexity of ...
3
votes
0answers
45 views

Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
3
votes
0answers
49 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
2
votes
0answers
50 views

How does back propagation adjust the hidden layers' weights and biases?

I'm new to neural networks and trying to figure out its fundamentals but I cannot fully understand the back propagation algorithm. In back propagation, I understand we want to go backwards from the ...
2
votes
0answers
98 views

Bias gradient of layer before batch normalization always zero

From the original paper and this post we have that batch normalization backpropagation can be formulated as I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
2
votes
0answers
71 views

Why should the weight updates be proportional to input?

I'm reading the book Grokking Deep Learning. Regarding weight updates during training, it has the following code and explanation: ...
2
votes
3answers
268 views

Backpropagation of neural nets with shared weight

I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. ...
2
votes
0answers
205 views

What is symbol-to-number differentiation?

I recently came across symbol-to-symbol and symbol-to-number differentiation, out of which symbol to symbol seemed fairly straightforward - the computational graph is extended to include gradient ...
2
votes
0answers
68 views

Why gradients are so small in deep learning?

The learning rate in my model is 0.00001 and the gradients of the model is within the distribution of [-0.0001, 0.0001]. Is it ...
2
votes
0answers
47 views

How is the gradient with respect to weights derived in batch normalization?

At the bottom of page 2 of the paper L2 Regularization versus Batch and Weight Normalization, the equation for the gradient of the output with respect to the weights is given as: $$ \triangledown y_{...
2
votes
0answers
15 views

yolo output and how to define labels for backpropogation on it

I want to build the yolo architecture in keras but can't understand the basic idea behind the training of the yolo, like how to define the labels for whether there is no object there what we have to ...
2
votes
0answers
36 views

Calculation of Neural network biases in backpropagation

While learning neural networks I've found a basic Python working example to play with. It has 3 input nodes, 4 nodes in a hidden layer, 1 output node. 5 data sets for training. The initial code is ...
2
votes
1answer
276 views

Training the generator in a GAN pair with back propagation

For the purposes of this question I am asking about training the generator, assume that training the discriminator is another topic. My understanding of generative adversarial networks is that you ...
2
votes
0answers
36 views

Use of backpropagation for weight updates in a combination of 2 neural networks

Every neural network updates its weights through back-propagation. How is back-propagation used for updating weights in a combination of 2 or more neural networks (e.g.:CNN-LSTM, GAN-CNN, etc.). For ...
2
votes
0answers
52 views

Using features extracted from a CNN as convolutional filter

I'm a bit confused about this. Assume I have a CNN network with two branches: Top Bottom The top branch outputs a feature vector of shape 1x1x1x10 (batch, h, w, c) The bottom branch outputs a ...
2
votes
1answer
3k views

How do I calculate the gradient of the hinge loss function?

With reference to the research paper entitled Sentiment Embeddings with Applications to Sentiment Analysis, I am trying to implement its sentiment ranking model in Python, for which I am required to ...
2
votes
0answers
156 views

Data prepared to linear regression. Can I use it with backpropagation?

I'm studying a Master's Degree in Artificial Intelligence and I need to learn how to use the Java Neural Network Simulator, JavaNNS, program. In one practice I have to build a neural network to use ...
2
votes
0answers
47 views

Recommendations on which architecture to use to guess appointment

I'm currently developping an application which allows psychologists to manage their schedule and budget. As a proof of concept, I would like to create an intelligent appointment service. There can be ...
2
votes
1answer
132 views

How to perform back-propagation in Decoupled Neural Interfaces?

I am attempting to create a fully decoupled feed-forward neural network by using decoupled neural interfaces (DNIs) as explained in the paper Decoupled Neural Interfaces using Synthetic Gradients (...
1
vote
0answers
25 views

CNN: Difficulties understanding backward pass derivatives

I have really quite hard difficulties to understand what is actually going on in the backward pass of a CNN. I am currently focusing on these references: https://towardsdatascience.com/forward-and-...
1
vote
0answers
40 views

How does backpropagation know which weights to change?

I'm currently working on constructing a neural network from scratch (in JavaScript). I'm in the middle of working on the backpropagation, but there's something I don't understand: how does the ...
1
vote
0answers
24 views

ReLU function converging to local optimum in one case and diverging in the other one

I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function. To check my implementation I ...
1
vote
0answers
48 views

Why doesn't anyone use reinforcement learning to find the best possible alternative to backpropagation?

To be clear, I'm very uninformed on the topic of alternative learning algorithms to backprop, all my knowledge comes from articles like these: lets-not-stop-at-backprop backprop-alternatives we-need-a-...
1
vote
0answers
47 views

Backpropagation not working as expected

I'm new to neural networks and I try to make a model that is guessing if a point is below or above relative to a function output. The idea is inspired from this video https://youtu.be/DGxIcDjPzac . ...
1
vote
0answers
156 views

Convolutional Layer Multichannel Backpropagation Implementation

I have been working on coding a CNN in python from scratch using numpy as a semester project and I think I have successfully implemented it up to backpropagation in the MaxPool Layers. However, my ...
1
vote
0answers
20 views

In a convolutional neural network, how is the error delta propagated between convolutional layers?

I'm coding some stuff for CNNs, just relying on numpy (and scipy just for the convolution operation for pure performance reasons). I've coded a small network consisting of a convolutional layer with ...
1
vote
0answers
54 views

Why is Openai's PPO2 implementation differentiable?

I'm trying to understand the concept behind the implementation of the OpenAI PPO2 algorithm. The loss function that is minimized is as follows: ...
1
vote
0answers
31 views

How do gradients are flown back into the Siamese network when branching is done?

I am curious about the working of a Siamese network. So, let us suppose I am using a triplet loss for my network and I have instantiated single CNN 3 times and there are 3 inputs to the network. So, ...
1
vote
0answers
23 views

For the generalised delta rule in back-propogation, do you subtract the target from the obtained output, or vice versa?

When I look up the generalised delta rule equation for back-propogation, I am seeing two conflicting equations. For example, here (slide 20), given $o$ (the output, defined in slide 18), $z$ (the ...
1
vote
0answers
34 views

How do I infer exploding or vanishing gradients in Keras?

It may already be obvious that I am just a practitioner and just a beginner to Deep Learning. I am still figuring out lots of "WHY"s and "HOW"s of DL. So, for example, if I train a ...
1
vote
0answers
52 views

How am I supposed to code equation 4.57 from the book "Machine Learning: An Algorithmic Perspective"?

Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained $$\delta_o(\kappa) = (y_\kappa -...
1
vote
0answers
66 views

XOR problem with bipolar representation

I am taking a course in Machine Learning and the Professor introduced us to the XOR problem. I understand the XOR problem is not linearly separable and we need to employ Neural Network for this ...
1
vote
0answers
38 views

Linear output layer back propagation

So I'm stack to something that it's probably very easy but I can't get my head around it. I'm building a Neural Network that will consist of many layers with non-linear activation functions (probably ...
1
vote
1answer
97 views

Computation of initial adjoint for NODE

I'm reading the paper Neural Ordinary Differential Equations and I have a simple question about adjoint method. When we train NODE, it uses a blackbox ODESolver to compute gradients through model ...
1
vote
0answers
92 views

How are weight matrices in attention learned?

I have been looking into transformers lately and have been reading tons of tutorials. All of them address the intuition behind attention, which I understand. However, they treat learning the weight ...
1
vote
0answers
64 views

I need help understanding general back propagation algorithm

In section 6.5.6 of the book Deep Learning by Ian et. al. general backpropagation algorithm is described as: The back-propagation algorithm is very simple. To compute the gradient of some scalar z ...
1
vote
0answers
254 views

Backward pass of CNN like Resnet: how to manually compute flops during backprop?

I've been trying to figure out how to compute the number of Flops in backward pass of ResNet. For forward pass, it seems straightforward: apply the conv filters to the input for each layer. But how ...
1
vote
0answers
28 views

Class of functional equations that backpropagation can solve

There is a theorem that states that basically a neural network can approximate any function whatsoever. However, this does not mean that it can solve any equation. I have some notes where it states ...
1
vote
0answers
38 views

Why do DeconvNet use ReLU in the backward pass?

Why does DeconvNet (Zeiler, 2014) use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply ...
1
vote
0answers
151 views

Which activation functions can lead to the vanishing gradient problem?

From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem. What other activation functions can lead to the ...
1
vote
0answers
37 views

Function to update weights in back-propagation

I am trying to wrap my head around how weights get updated during back propagation. I've been going through a school book and I have the following setup for an ANN with 1 hidden layer, a couple of ...
1
vote
0answers
140 views

Is this TensorFlow implementation of partial derivative of the cost with respect to the bias correct?

I have a neural network for MNIST classification which I am hard coding using TensorFlow 2.0. The neural network has an input layer consisting of 784 neurons (28 * 28), one hidden layer having "...
1
vote
0answers
76 views

Understanding the partial derivative with respect to the weight matrix and bias

Say we have the layer $X W + b = Y$. I want to get $\frac{dL}{dW}$ and we assume I have $\frac{dL}{dY}$. So all I need is to find $\frac{dY}{dW}$. I know that it should be $X^T\frac{dL}{dY}$ but don'...
1
vote
0answers
58 views

How to train and update weights of filters

I have some problems with training CNN :( For example: Input 6x6x3, 1 core 3x3x3, output = 4x4x1 => pool: 2x2x1 By backpropagation I calculated deltas for output. This tutor and other tutors are ...
1
vote
2answers
154 views

Which neuron represents which part of the input?

In a neural network, each neuron represents some part of the input. For example, in the case of a MNIST digit, consider the stem of the number 9. Each neuron in the NN represents some part of this ...