Questions tagged [backpropagation]
For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.
234
questions
34
votes
4answers
32k views
What is the time complexity for training a neural network using back-propagation?
Suppose that a NN contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_i$ nodes in each layer. What is the time complexity to train this NN using back-propagation?
I have a basic ...
18
votes
1answer
454 views
Are these two versions of back-propagation equivalent?
Just for fun, I am trying to develop a neural network.
Now, for backpropagation I saw two techniques.
The first one is used here and in many other places too.
What it does is:
It computes the ...
13
votes
2answers
4k views
Is the mean-squared error always convex in the context of neural networks? [duplicate]
Multiple resources I referred to mention that MSE is great because it's convex. But I don't get how, especially in the context of neural networks.
Let's say we have the following:
$X$: training ...
10
votes
5answers
625 views
What is "backprop"?
What does "backprop" mean? Is the "backprop" term basically the same as "backpropagation" or does it have a different meaning?
10
votes
2answers
612 views
What are the learning limitations of neural networks trained with backpropagation?
In 1969, Seymour Papert and Marvin Minsky showed that Perceptrons could not learn the XOR function.
This was solved by the backpropagation network with at least one hidden layer. This type of network ...
9
votes
2answers
155 views
How do evolutionary algorithms have advantages over the conventional backpropagation methods?
How does employing evolutionary algorithms to design and train artificial neural networks have advantages over using the conventional backpropagation algorithms?
7
votes
2answers
5k views
How to combine backpropagation in neural nets and reinforcement learning?
I have followed a course on machine learning, where we learned about the gradient descent (GD) and back-propagation (BP) algorithms, which can be used to update the weights of neural networks, and ...
7
votes
3answers
898 views
How do I know if my backpropagation is implemented correctly?
I'm working on implementation of the backpropagation algorithm for a simple neural network which predicts a probability of survival (1 or 0) and I can't get it above 80% no matter how much I try to ...
7
votes
3answers
2k views
CNN backpropagation with stride>1
I read that to compute the derivative of the error with respect to the input of a convolution layer is the same to make of a convolution between deltas of the next layer and the weight matrix rotated ...
7
votes
1answer
262 views
What, exactly, does the REINFORCE update equation mean?
I understand that this is the update for the parameters of a policy in REINFORCE:
$$
\Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}
$$
Where 𝑣𝑡 is ...
7
votes
1answer
7k views
How is the gradient calculated for the middle layer's weights?
I am trying to understand backpropagation. I used a simple neural network with one input $x$, one hidden layer $h$ and one output layer $y$, with weight $w_1$ connecting $x$ to $h$, and $w_2$ ...
7
votes
2answers
90 views
Why do very deep non resnet architectures perform worse compared to shallower ones for the same iteration? Shouldn't they just train slower?
My understanding of the vanishing gradient problem in deep networks is that as backprop progresses through the layers the gradients become small, and thus training progresses slower. I'm having a hard ...
6
votes
1answer
4k views
How to avoid falling into the "local minima" trap?
How do I avoid my gradient descent algorithm into falling into the "local minima" trap while backpropogating on my neural network?
Are there any methods which help me avoid it?
6
votes
1answer
402 views
Why not teach to a NN not only what is true, but also what is not true?
I'm not a person who studies neural networks, or does anything that is related with that area, but I have seen a couple of seminars, videos (such as 3Blue1Brown's Series), and what I am always told is ...
6
votes
2answers
990 views
In deep learning, is it possible to use discontinuous activation functions?
In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)?
(My guess: for example, ReLU is non-differentiable at a single point, but it still has a ...
6
votes
1answer
321 views
What kind of algorithm is the Levenberg–Marquardt algorithm?
Is a Levenberg–Marquardt algorithm a type of back-propagation algorithm or is it a different category of algorithm?
Wikipedia says that it is a curve fitting algorithm. How is a curve fitting ...
6
votes
1answer
149 views
Why did the developement of neural networks stop between 50s and 80s?
In a video lecture on the development of neural networks and the history of deep learning (you can start from minute 13), the lecturer (Yann LeCunn) said that the development of neural networks ...
6
votes
0answers
167 views
How to design 4D Deep Recurrent Neural Networks using Tensorflow?
I want to design a simple model that predicts the movement of coordinates with RNNs.
In a typical three-dimensional LSTM model, one feature is encoded as one hot encoding, and the ...
6
votes
2answers
1k views
How do we design a neural network such that the $L_1$ norm of the outputs is less than or equal to 1?
What are some ways to design a neural network with the restriction that the $L_1$ norm of the output values must be less than or equal to 1? In particular, how would I go about performing back-...
5
votes
3answers
1k views
What is the actual learning algorithm: back-propagation or gradient descent?
What is the actual learning algorithm: back-propagation or gradient descent (or, in general, the optimization algorithm)?
I am reading through chapter 8 of Parallel Distributed Processing hand book ...
5
votes
2answers
884 views
What exactly is averaged when doing batch gradient descent?
I have a question about how the averaging works when doing mini-batch gradient descent.
I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-...
5
votes
2answers
929 views
What is the purpose of the batch size in neural networks?
Why is a batch size needed to update the weights of a neural network?
According to that Youtube Video from 3B1B, the weights are updated by calculating the error between expectation and outcome of ...
5
votes
2answers
223 views
Why is tf.abs non-differentiable in Tensorflow?
I understand why tf.abs is non-differentiable in principle (discontinuity at 0) but the same applies to tf.nn.relu yet, in case of this function gradient is simply set to 0 at 0. Why the same logic is ...
5
votes
2answers
96 views
Is the gradient at a layer independent of the activations of the previous layers?
Is the gradient at a layer (of a feed-forward neural network) independent of the activations of the previous layers?
I read this in a paper titled Mean Field Residual Networks: On the Edge of Chaos (...
5
votes
2answers
115 views
Are on-line backpropagation iterations perpendicular to the constraint?
Raul Rojas' Neural Networks A Systematic Introduction, section 8.1.2 relates off-line backpropagation and on-line backpropagation with Gauss-Jacobi and Gauss-Seidel methods for finding the ...
5
votes
1answer
3k views
How to train a CNN
When it comes to CNNs, I don't understand 2 things in the training process:
How do I pass the error back when there are pooling layers between the convolutional layers?
And if I know how it's done, ...
5
votes
1answer
588 views
What makes learned feature detectors specialize in CNN?
It has been shown that it is possible to use unsupervised learning techniques to produce good feature detectors in CNNs. I can't understand what drives specialization of those feature detectors. In ...
5
votes
1answer
175 views
What would an implementation of this Neural Network look like?
I'm relatively new to neural networks and was wondering what an implementation of this paper would look like. More specifically, how are the correct values of Kp, Ki, and Kd determined at run time so ...
5
votes
1answer
103 views
Why is the change in cost wrt bias in neural network equal to error in the neuron?
While reading the book on neural networks by Michael Nielson, I had a problem understanding equation (BP3), which is
$$
\frac{\partial C}{\partial b_{j}^{l}}=\delta_{j}^{l} \tag{BP3}\label{BP3},
$$
...
5
votes
0answers
47 views
Why does sigmoid saturation prevent signal flow through the neuron?
As per these slides on page 35:
Sigmoids saturate and kill gradients.
when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero.
the gradient and ...
4
votes
3answers
3k views
How to test if my implementation of back propagation neural Network is correct
I am working on an implementation of the back propagation algorithm. What I have implemented so far seems working but I can't be sure that the algorithm is well implemented, here is what I have ...
4
votes
1answer
1k views
Do we know what the units of neural networks will do before we train them?
I was learning about back-propagation and, looking at the algorithm, there is no particular 'partiality' given to any unit. What I mean by partiality there is that you have no particular ...
4
votes
1answer
2k views
Is back-propagation applied for each data point or for a batch of data points?
I am new to deep learning and trying to understand the concept of back-propagation. I have a doubt about when the back-propagation is applied. Assume that I have a training data set of 1000 images ...
4
votes
2answers
72 views
Could error surface shape be useful to detect which local minima is better for generalization?
The following plot shows error function output based on system weights.
Two equal local minima are shown in green pointers. Note that the red dots are not related to the question.
Does the right one ...
4
votes
2answers
161 views
Are filters fixed or learned?
No matter what I google or what paper I read, I can't find an answer to my question. In a deep convolutional neural network, let's say AlexNet (Krizhevsky, 2012), filters' weights are learned by means ...
4
votes
1answer
250 views
Finding an optimum back propagation algorithm
I recently started working on very simple machine learning codes in Python and I came across a big problem: teaching the system to improve on its guesses.
So this is what the code is about:
I will ...
4
votes
1answer
123 views
Is the bias also a "weight" in a neural network?
I'm learning about how neural networks are trained. I understand how a neuron works, backpropagation, and all that. In neurons, there is a clear distinction between a "weight" and a "...
4
votes
1answer
87 views
How can I use one neural network for both players in Alpha Zero (Connect 4)?
First of all, it is great to have found this community!
I am currently implementing my own Alpha Zero clone on Connect4. However, I have a mental barrier I cannot overcome.
How can I use one neural ...
4
votes
1answer
379 views
Does training happen during NEAT?
When one uses NEAT to evolve the best fitting network for a task, does training take place in each epoch as well?
If I understand correctly, training is the adjustment of the weights of the neural ...
4
votes
1answer
128 views
What are some concrete steps to deal with the vanishing gradient problem?
I am training an ANN for classification between 3 classes. The ANN has an input layer, one hidden layer and a 3 node output layer.
The problem I am facing is that the output being produced by the 3 ...
4
votes
1answer
555 views
What should I do with the flatten layer during back-propagation? [duplicate]
I'm creating a CNN network without other frameworks such as PyTorch, Keras, Tensorflow, and so on.
During the forward pass, the Flatten layer reshapes the previous ...
4
votes
1answer
150 views
How to perform back propagation with different sized layers?
I'm developing my first neural network, using the well known MNIST database of handwritten digit. I want the NN to be able to classify a number from 0 to 9 given an image.
My neural network consists ...
4
votes
1answer
186 views
How are filters weights updated for a CNN?
I've been trying to learn backpropagation for CNNs. I read several articles like this one and this one. They all say that to compute the gradients for the filters, you just do a convolution with the ...
4
votes
1answer
47 views
Am I able to visualize the differentiation in backprop as follows?
I'm wondering if I can visualize the backprop process as follows (please excuse me if I have written something terrible wrong). If the loss function $L$ on a neural network represents the function has ...
4
votes
2answers
101 views
How can I implement back-propagation for medium-sized neural networks?
I've been wanting to make my own Neural Network in Python, in order to better understand how it works. I've been following this series of videos as a sort of guide, but it seems the backpropagation ...
4
votes
1answer
380 views
In the case of invalid actions, which output probability matrix should we use in back-propagation?
As discussed in this thread, you can handle invalid moves in reinforcement learning by re-setting the probabilities of all illegal moves to zero and renormalising the output vector.
In back-...
4
votes
1answer
158 views
Can a neural network learn to avoid wrong decisions using backpropagation?
I studied the articles on Neural Networks and Deep Learning from Michael Nielsen and developed a simple neural network based on his examples. I understand how backpropagation works and I already ...
4
votes
1answer
147 views
Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?
I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation.
As per my knowledge, ...
4
votes
1answer
430 views
How can a DQN backpropagate its loss?
I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I ...
4
votes
1answer
102 views
Backpropagation equation for a variant on the usual Linear Neuron architecture
Recently I encountered a variant on the normal linear neural layer architecture: Instead of $Z = XW + B$, we now have $Z = (X-A)W + B$. So we have a 'pre-bias' $A$ that affects the activation of the ...
