Questions tagged [backpropagation]

For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.

77 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
0answers
162 views

How to design 4D Deep Recurrent Neural Networks using Tensorflow?

I want to design a simple model that predicts the movement of coordinates with RNNs. In a typical three-dimensional LSTM model, one feature is encoded as one hot encoding, and the ...
4
votes
0answers
28 views

How do weights changes handles during back-propagation when there are unknown labels

I have a question about how weights are updated during back-propagation for some of my samples that have unknown labels (please note, unknown, not missing). The reason they are unknown is because this ...
4
votes
1answer
96 views

Backpropagation equation for a variant on the usual Linear Neuron architecture

Recently I encountered a variant on the normal linear neural layer architecture: Instead of $Z = XW + B$, we now have $Z = (X-A)W + B$. So we have a 'pre-bias' $A$ that affects the activation of the ...
3
votes
0answers
41 views

Why does sigmoid saturation prevent signal flow through the neuron?

As per these slides on page 35: Sigmoids saturate and kill gradients. when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero. the gradient and ...
3
votes
1answer
97 views

What is asymmetric relaxation backpropagation?

In Chapter 8, section 8.5.2, Raul Rojas describes how the weights for a layer of a neural network can be calculated using a pseudoinverse of the sigmoid function in the nodes, he explains this is an ...
3
votes
0answers
83 views

What is the time complexity for training a gated recurrent unit (GRU) neural network using back-propagation through time?

Let us assume we have a GRU network containing $H$ layers to process a training dataset with $K$ tuples, $I$ features, and $H_i$ nodes in each layer. I have a pretty basic idea how the complexity of ...
3
votes
0answers
92 views

Doing backpropagation in an Tensorflow.js Neural Network

I have a neural network (which I am making from scratch). In order to make the neural network "learn" I need to conduct back-propagation. Using the code at the below how would I conduct back-...
3
votes
0answers
43 views

Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
3
votes
0answers
45 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
3
votes
0answers
92 views

How can I formulate a nonogram problem as a constraint satisfaction problem?

I've just started learning CSP and I find it quite exciting. Now I'm facing nonogram solving problem and I want to solve it using backtracking with CSP. The first problem that I face is that I cannot ...
3
votes
0answers
706 views

How to calculate gradient of filter in convolution network

I have similar architecture like in image:CNN. I don't understand how to calculate gradient of filter F. I found these equations(source): Gradient and delta, where first equation calculate gradient ...
3
votes
1answer
119 views

How to perform back-propagation in Decoupled Neural Interfaces?

I am attempting to create a fully decoupled feed-forward neural network by using decoupled neural interfaces (DNIs) as explained in the paper Decoupled Neural Interfaces using Synthetic Gradients (...
3
votes
1answer
387 views

How does backpropagation work on a custom loss function whose components have magnitudes of different orders?

I want to use a custom loss function which is a weighted combination of l1 and DSSIM losses. The DSSIM loss is limited between 0 and 0.5 where as the l1 loss can be orders of magnitude greater and is ...
2
votes
0answers
69 views

Bias gradient of layer before batch normalization always zero

From the original paper and this post we have that batch normalization backpropagation can be formulated as I'm interested in the derivative of the previous layer outputs $x_i=\sigma(w X_i+b)$ with ...
2
votes
0answers
64 views

Why should the weight updates be proportional to input?

I'm reading the book Grokking Deep Learning. Regarding weight updates during training, it has the following code and explanation: ...
2
votes
3answers
150 views

Backpropagation of neural nets with shared weight

I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. ...
2
votes
0answers
130 views

What is symbol-to-number differentiation?

I recently came across symbol-to-symbol and symbol-to-number differentiation, out of which symbol to symbol seemed fairly straightforward - the computational graph is extended to include gradient ...
2
votes
0answers
65 views

Why gradients are so small in deep learning?

The learning rate in my model is 0.00001 and the gradients of the model is within the distribution of [-0.0001, 0.0001]. Is it ...
2
votes
0answers
39 views

How is the gradient with respect to weights derived in batch normalization?

This is a cross-post, as I didn't get any answers on Stats SE and I am hoping that it gets more attention here. At the bottom of page 2 of the paper L2 Regularization versus Batch and Weight ...
2
votes
0answers
14 views

yolo output and how to define labels for backpropogation on it

I want to build the yolo architecture in keras but can't understand the basic idea behind the training of the yolo, like how to define the labels for whether there is no object there what we have to ...
2
votes
0answers
28 views

Calculation of Neural network biases in backpropagation

While learning neural networks I've found a basic Python working example to play with. It has 3 input nodes, 4 nodes in a hidden layer, 1 output node. 5 data sets for training. The initial code is ...
2
votes
0answers
42 views

How does a Bidirectional RNN work?

Could it be possible to reach a similar output via feeding a unidirectional network with the original data and the data played backwards?
2
votes
2answers
121 views

Which neuron represents which part of the input?

In a neural network, each neuron represents some part of the input. For example, in the case of a MNIST digit, consider the stem of the number 9. Each neuron in the NN represents some part of this ...
2
votes
1answer
212 views

Training the generator in a GAN pair with back propagation

For the purposes of this question I am asking about training the generator, assume that training the discriminator is another topic. My understanding of generative adversarial networks is that you ...
2
votes
0answers
35 views

Use of backpropagation for weight updates in a combination of 2 neural networks

Every neural network updates its weights through back-propagation. How is back-propagation used for updating weights in a combination of 2 or more neural networks (e.g.:CNN-LSTM, GAN-CNN, etc.). For ...
2
votes
0answers
110 views

Update of weights in Recurrent Neural Network through back propagation

How does Recurrent Neural Network updates its weights and bias through backpropagation? Is time taken into account while updating the weights of a RNN using Backpropagation through time(BPTT)?"
2
votes
0answers
52 views

Using features extracted from a CNN as convolutional filter

I'm a bit confused about this. Assume I have a CNN network with two branches: Top Bottom The top branch outputs a feature vector of shape 1x1x1x10 (batch, h, w, c) The bottom branch outputs a ...
2
votes
0answers
142 views

Data prepared to linear regression. Can I use it with backpropagation?

I'm studying a Master's Degree in Artificial Intelligence and I need to learn how to use the Java Neural Network Simulator, JavaNNS, program. In one practice I have to build a neural network to use ...
2
votes
0answers
47 views

Recommendations on which architecture to use to guess appointment

I'm currently developping an application which allows psychologists to manage their schedule and budget. As a proof of concept, I would like to create an intelligent appointment service. There can be ...
1
vote
0answers
13 views

In a convolutional neural network, how is the error delta propagated between convolutional layers?

I'm coding some stuff for CNNs, just relying on numpy (and scipy just for the convolution operation for pure performance reasons). I've coded a small network consisting of a convolutional layer with ...
1
vote
2answers
45 views

How can I model any structure for a neural network?

Hello I am currently doing research on the effect of altering a neural network's structure. Particularly I am investigating what affect would putting a random DAG (directed acyclic graph) in the ...
1
vote
0answers
48 views

Why is Openai's PPO2 implementation differentiable?

I'm trying to understand the concept behind the implementation of the OpenAI PPO2 algorithm. The loss function that is minimized is as follows: ...
1
vote
0answers
27 views

How do gradients are flown back into the Siamese network when branching is done?

I am curious about the working of a Siamese network. So, let us suppose I am using a triplet loss for my network and I have instantiated single CNN 3 times and there are 3 inputs to the network. So, ...
1
vote
0answers
17 views

For the generalised delta rule in back-propogation, do you subtract the target from the obtained output, or vice versa?

When I look up the generalised delta rule equation for back-propogation, I am seeing two conflicting equations. For example, here (slide 20), given $o$ (the output, defined in slide 18), $z$ (the ...
1
vote
0answers
24 views

How do I infer exploding or vanishing gradients in Keras?

It may already be obvious that I am just a practitioner and just a beginner to Deep Learning. I am still figuring out lots of "WHY"s and "HOW"s of DL. So, for example, if I train a ...
1
vote
0answers
41 views

How am I supposed to code equation 4.57 from the book “Machine Learning: An Algorithmic Perspective”?

Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained $$\delta_o(\kappa) = (y_\kappa -...
1
vote
0answers
31 views

Linear output layer back propagation

So I'm stack to something that it's probably very easy but I can't get my head around it. I'm building a Neural Network that will consist of many layers with non-linear activation functions (probably ...
1
vote
1answer
68 views

Computation of initial adjoint for NODE

I'm reading the paper Neural Ordinary Differential Equations and I have a simple question about adjoint method. When we train NODE, it uses a blackbox ODESolver to compute gradients through model ...
1
vote
0answers
56 views

How are weight matrices in attention trained?

I have been looking into transformers lately and been reading tons of tutorials. All of them address the intuition behind attention, which I understand, but they treat training the weight matrices for ...
1
vote
0answers
60 views

I need help understanding general back propagation algorithm

In section 6.5.6 of the book Deep Learning by Ian et. al. general backpropagation algorithm is described as: The back-propagation algorithm is very simple. To compute the gradient of some scalar z ...
1
vote
0answers
179 views

Backward pass of CNN like Resnet: how to manually compute flops during backprop?

I've been trying to figure out how to compute the number of Flops in backward pass of ResNet. For forward pass, it seems straightforward: apply the conv filters to the input for each layer. But how ...
1
vote
0answers
25 views

Class of functional equations that backpropagation can solve

There is a theorem that states that basically a neural network can approximate any function whatsoever. However, this does not mean that it can solve any equation. I have some notes where it states ...
1
vote
0answers
36 views

Why do DeconvNet use ReLU in the backward pass?

Why does DeconvNet (Zeiler, 2014) use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply ...
1
vote
0answers
78 views

Which activation functions can lead to the vanishing gradient problem?

From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem. What other activation functions can lead to the ...
1
vote
0answers
33 views

Function to update weights in back-propagation

I am trying to wrap my head around how weights get updated during back propagation. I've been going through a school book and I have the following setup for an ANN with 1 hidden layer, a couple of ...
1
vote
0answers
106 views

Is this TensorFlow implementation of partial derivative of the cost with respect to the bias correct?

I have a neural network for MNIST classification which I am hard coding using TensorFlow 2.0. The neural network has an input layer consisting of 784 neurons (28 * 28), one hidden layer having "...
1
vote
0answers
51 views

Understanding the partial derivative with respect to the weight matrix and bias

Say we have the layer $X W + b = Y$. I want to get $\frac{dL}{dW}$ and we assume I have $\frac{dL}{dY}$. So all I need is to find $\frac{dY}{dW}$. I know that it should be $X^T\frac{dL}{dY}$ but don'...
1
vote
1answer
53 views

How does a single neuron in hidden layer affect training accuracy

I'm currently a student learning about AI Networks. I've came across a statement in one of my Professor's books that a FFBP (Feed-Forward Back-Propagation) Neural Network with a single hidden layer ...
1
vote
0answers
55 views

How to train and update weights of filters

I have some problems with training CNN :( For example: Input 6x6x3, 1 core 3x3x3, output = 4x4x1 => pool: 2x2x1 By backpropagation I calculated deltas for output. This tutor and other tutors are ...
1
vote
0answers
201 views

Difficulties to implement the layer-wise relevance propagation in MATLAB

I'm having serious issues with the implementation of the LRP algorithm for neural networks in MATLAB. The challenge is to implement the equations correctly. I'm trying to implement the deep-Taylor $\...