# Questions tagged [backpropagation]

For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.

180 questions
Filter by
Sorted by
Tagged with
698 views

### Is back-propagation applied for each data point or for a batch of data points?

I am new to deep learning and trying to understand the concept of back-propagation. I have a doubt about when the back-propagation is applied. Assume that I have a training data set of 1000 images ...
187 views

### What exactly is averaged when doing batch gradient descent?

I have a question about how the averaging works when doing mini-batch gradient descent. I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-...
37 views

### Computation of initial adjoint for NODE

I'm reading the paper Neural Ordinary Differential Equations and I have a simple question about adjoint method. When we train NODE, it uses a blackbox ODESolver to compute gradients through model ...
32 views

### How does a single neuron in hidden layer affect training accuracy

I'm currently a student learning about AI Networks. I've came across a statement in one of my Professor's books that a FFBP (Feed-Forward Back-Propagation) Neural Network with a single hidden layer ...
674 views

### Does backpropagation update weights one layer at a time?

I am new to Deep Learning. Suppose that we have a neural network with one input layer, one output layer, and one hidden layer. Let's refer to the weights from input to hidden as $W$ and the weights ...
355 views

### In the case of invalid actions, which output probability matrix should we use in back-propagation?

As discussed in this thread, you can handle invalid moves in reinforcement learning by re-setting the probabilities of all illegal moves to zero and renormalising the output vector. In back-...
52 views

### Why should the weight updates be proportional to input?

I'm reading the book Grokking Deep Learning. Regarding weight updates during training, it has the following code and explanation: ...
555 views

### What are the learning limitations of neural networks trained with backpropagation?

In 1969, Seymour Papert and Marvin Minsky showed that Perceptrons could not learn the XOR function. This was solved by the backpropagation network with at least one hidden layer. This type of network ...
24 views

### Positive bias causes the calculation of incorrect gradients

I have a data set with a positive bias (an image, where the values range from 0 to 1), that seems to be causing my network to calculate incorrect gradients. If I just use the raw image as input, of ...
22 views

### In CNNs, why do we sum the filter derivatives w.r.t the loss function to get the final gradient?

In a Convolutional Neural Network, unlike the fully connected layers, the same filter is used multiple times on the input while convolving - so during backpropagation, we get multiple derivatives for ...
358 views

### How does backpropagation work on a custom loss function whose components have magnitudes of different orders?

I want to use a custom loss function which is a weighted combination of l1 and DSSIM losses. The DSSIM loss is limited between 0 and 0.5 where as the l1 loss can be orders of magnitude greater and is ...
36 views

116 views

### What, exactly, does the REINFORCE update equation mean?

I understand that this is the update for the parameters of a policy in REINFORCE: $$\Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}$$ Where 𝑣𝑡 is ...
24 views

### Back propagation approach to logistic regression: why is cost diverging but accuracy increasing?

Background I have tried to fit a logistic regression model - written using a forward / back propagation approach (as part of Andrew Ng's deep learning course) - to a very non-linear data set (see ...
135 views

### What should I do with the flatten layer during back-propagation?

I'm creating a CNN network without other frameworks such as PyTorch, Keras, Tensorflow, and so on. During the forward pass, the Flatten layer reshapes the previous ...
42 views

### What are the rules behind vector product in gradient?

Let's suppose we have calculated the gradient and it came out to be $f(WX)(1-f(W X))X$, where $f()$ is the sigmoid function, $W$ of order $2\times2$ is the weight matrix, and $X$ is an input vector of ...
41 views

### How to normalise image input to backpropogation algorithm?

I am implementing a simple backpropagation neural network for classifying images. One set of images are cars another set of images are buildings (houses). So far I have used Sobel Edge detector after ...
116 views

### Why does a neuron in a multi-layer network need several input connections?

For example, if I have the following architecture: Each neuron in the hidden layer has a connection from each one in the input layer. 3 x 1 Input Matrix and a 4 x 3 weight matrix (for the ...
128 views

### Simple three layer neural network with backpropagation is not approximating tanh function

I have this simple neural network in Python which I'm trying to use to aproximation tanh function. As inputs I have x - inputs to the function, and as outputs I want tanh(x) = y. I'm using sigmoid ...
27 views

### Can the normal equation be used to optimise the RNN's weights?

I have made an RNN from scratch in Tensorflow.js. In order to update my weights (without needing to calculate the derivatives), I thought of using the normal equation to find the optimal values for my ...
80 views

### Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?

I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation. As per my knowledge, ...
70 views

### Doing backpropagation in an Tensorflow.js Neural Network

I have a neural network (which I am making from scratch). In order to make the neural network "learn" I need to conduct back-propagation. Using the code at the below how would I conduct back-...
283 views

### What is the proof behind the gradient of a curve being proportional to the distance between the two co-ordinates in the x-axis?

In the [delta rule] the equation to adjust the weight with respect to error is $$w_{(n+1)}=w_{(n)}-\alpha \times \frac{\partial E}{\partial w}$$ *where $\alpha$ is the learning rate and $E$ is the ...
38 views

### Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
94 views

### Backpropagation equation for a variant on the usual Linear Neuron architecture

Recently I encountered a variant on the normal linear neural layer architecture: Instead of $Z = XW + B$, we now have $Z = (X-A)W + B$. So we have a 'pre-bias' $A$ that affects the activation of the ...
26 views

### How to find the derivative of a dynamic neuron model, which depends on previous states of the neuron?

This is the equation where n denotes the current state, (n-1) denotes the state in the previous step etc. And to do back-propagation I need to find partial derivatives over each of the variables. For ...
22 views

### Finding the energy function given update rule of a single layer non-linear neural network

Consider the network with N neurons, each of which takes a $2 \times k$ input specified by the tuple $(\vec c_t, \vec \theta_t)$ to produce output $\vec{R}_t$ through an update rule on the pairwise ...
84 views

### Why do very deep non resnet architectures perform worse compared to shallower ones for the same iteration? Shouldn't they just train slower?

My understanding of the vanishing gradient problem in deep networks is that as backprop progresses through the layers the gradients become small, and thus training progresses slower. I'm having a hard ...
58 views

### Backpropagation implementation with Java

I've been trying to implement a Multilayer Perceptron Network using java language with the ultimate goal of creating and teaching a neural network to recognize handwritten digits. Pretty simple and ...
41 views

### Why is it that having a duplicate in features set makes training to work bad

I'm defining a deep network to emulate a multitarget regression. When I costruct my training set, I take information from a graph; without going into too much detail, it could happen that I take 2 ...
65 views

### Is my understanding of back-propogation correct?

I am trying to learn backpropagation and this is what I know so far. To update the weights of the neural network you have to figure out the partial derivative of each of the parameters on the loss ...
58 views

### I need help understanding general back propagation algorithm

In section 6.5.6 of the book Deep Learning by Ian et. al. general backpropagation algorithm is described as: The back-propagation algorithm is very simple. To compute the gradient of some scalar z ...
53 views

### How is a neural network where the majority of inputs are 0 trained?

Consider AlexNet, which has 1000 output nodes, each of which classifies an image: The problem I have been having with training a neural network of similar proportions, is that it does what any ...