Questions tagged [backpropagation]
For questions related to the technique of backpropagation, whereby the loss, error, or correction signal calculated at the output of an artificial network output is fed back to the parameters in each layer of the network until the network's behavior converges to a training state within the required accuracy and reliability.
100
questions
15
votes
1answer
354 views
Differences between backpropagation techniques
Just for fun, I am trying to develop a neural network.
Now, for backpropagation I saw two techniques.
The first one is used here and in many other places too.
What it does is:
It computes the ...
14
votes
2answers
8k views
What is the time complexity for training a neural network using back-propagation?
Suppose that a NN contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_i$ nodes in each layer. What is the time complexity to train this NN using back-propagation?
I have a basic ...
9
votes
2answers
584 views
Is the mean-squared error always convex in the context of neural networks?
Multiple resources I referred to mention that MSE is great because it's convex. But I don't get how, especially in the context of neural networks.
Let's say we have the following:
$X$: training ...
8
votes
3answers
406 views
What is “backprop”?
What does "backprop" mean? I've Googled it, but it's showing backpropagation.
Is the "backprop" term basically the same as "backpropagation" or does it have a different meaning?
8
votes
2answers
120 views
How do evolutionary algorithms have advantages over the conventional backpropagation methods?
How does employing evolutionary algorithms to design and train artificial neural networks have advantages over using the conventional backpropagation algorithms?
6
votes
3answers
475 views
How do I know if my backpropagation is implemented correctly?
I'm working on implementation of the backpropagation algorithm for a simple neural network which predicts a probability of survival (1 or 0) and I can't get it above 80% no matter how much I try to ...
6
votes
2answers
471 views
What are the limits to what can be learnt using a backpropagation neural network?
In 1969, Seymour Papert and Marvin Minsky showed that Perceptrons could not learn the XOR function.
This was solved by the backpropagation network with at least one hidden layer. This type of ...
6
votes
2answers
5k views
How is gradient calculated for middle layer weights?
I am trying to understand backpropagation. I used a simple neural network with one input x, one hidden layer h and one output layer y, with weight w1 connecting x to h, and w2 connecting h to y.
x--...
5
votes
1answer
189 views
Why not teach to a NN not only what is true, but also what is not true?
I'm not a person who studies neural networks, or does anything that is related with that area, but I have seen a couple of seminars, videos (such as 3Blue1Brown's Series), and what I am always told is ...
5
votes
1answer
3k views
How to avoid falling into the “local minima” trap?
How do I avoid my gradient descent algorithm into falling into the "local minima" trap while backpropogating on my neural network?
Are there any methods which help me avoid it?
5
votes
3answers
526 views
CNN backpropagation with stride>1
I read that to compute the derivative of the error with respect to the input of a convolution layer is the same to make of a convolution between deltas of the next layer and the weight matrix rotated ...
5
votes
2answers
103 views
Are on-line backpropagation iterations perpendicular to the constraint?
Raul Rojas' Neural Networks A Systematic Introduction, section 8.1.2 relates off-line backpropagation and on-line backpropagation with Gauss-Jacobi and Gauss-Seidel methods for finding the ...
5
votes
1answer
476 views
What makes learned feature detectors specialize in CNN?
It has been shown that it is possible to use unsupervised learning techniques to produce good feature detectors in CNNs. I can't understand what drives specialization of those feature detectors. In ...
5
votes
1answer
167 views
What would an implementation of this Neural Network look like?
I'm relatively new to neural networks and was wondering what an implementation of this paper would look like. More specifically, how are the correct values of Kp, Ki, and Kd determined at run time so ...
5
votes
1answer
282 views
What kind of algorithm is the Levenberg–Marquardt algorithm?
Is a Levenberg–Marquardt algorithm a type of back-propagation algorithm or is it a different category of algorithm?
Wikipedia says that it is a curve fitting algorithm. How is a curve fitting ...
5
votes
1answer
38 views
Why do very deep non resnet architectures perform worse compared to shallower ones for the same iteration? Shouldn't they just train slower?
My understanding of the vanishing gradient problem in deep networks is that as backprop progresses through the layers the gradients become small, and thus training progresses slower. I'm having a hard ...
5
votes
0answers
122 views
How to design 4D Deep Recurrent Neural Networks using Tensorflow?
I want to design a simple model that predicts the movement of coordinates with RNNs.
In a typical three-dimensional LSTM model, one feature is encoded as one hot encoding, and the ...
4
votes
3answers
2k views
How to test if my implementation of back propagation neural Network is correct
I am working on an implementation of the back propagation algorithm. What I have implemented so far seems working but I can't be sure that the algorithm is well implemented, here is what I have ...
4
votes
1answer
1k views
Do we know what the units of neural networks will do before we train them?
I apologize if this is a repeated question or if this is too simple. I was learning about back-propagation and looking at the algorithm there is no particular 'partiality' given to any unit. What I ...
4
votes
4answers
383 views
What is the actual learning algorithm: back-propagation or gradient descent?
What is the actual learning algorithm: back-propagation or gradient descent (or, in general, the optimization algorithm)?
I am reading through chapter 8 of Parallel Distributed Processing hand book ...
4
votes
1answer
197 views
Finding an optimum back propagation algorithm
I recently started working on very simple machine learning codes in Python and I came across a big problem: teaching the system to improve on its guesses.
So this is what the code is about:
I will ...
4
votes
2answers
610 views
Are Dreams a Form of Backpropagation?
Humans often dream of random events that occurred during the day. Could the reason for this be that our brains are backpropagating errors while we sleep, and we see the result of these ...
4
votes
1answer
43 views
How can I use one neural network for both players in Alpha Zero (Connect 4)?
First of all, it is great to have found this community!
I am currently implementing my own Alpha Zero clone on Connect4. However, I have a mental barrier I cannot overcome.
How can I use one neural ...
4
votes
1answer
100 views
What are some concrete steps to deal with the vanishing gradient problem?
I am training an ANN for classification between 3 classes. The ANN has an input layer, one hidden layer and a 3 node output layer.
The problem I am facing is that the output being produced by the 3 ...
4
votes
1answer
117 views
How are filters weights updated for a CNN?
I've been trying to learn backpropagation for CNNs. I read several articles like this one and this one. They all say that to compute the gradients for the filters, you just do a convolution with the ...
4
votes
1answer
40 views
Am I able to visualize the differentiation in backprop as follows?
I'm wondering if I can visualize the backprop process as follows (please excuse me if I have written something terrible wrong). If the loss function $L$ on a neural network represents the function has ...
4
votes
1answer
2k views
How to train a CNN
When it comes to CNNs, I don't understand 2 things in the training process:
How do I pass the error back when there are pooling layers between the convolutional layers?
And if I know how it's done, ...
4
votes
1answer
74 views
Backpropagation With Medium-sized Neural Networks
So, I've been wanting to make my own Neural Network in Python, in order to better understand how it works. I've been following this series of videos as a sort of guide, but it seems the ...
4
votes
1answer
145 views
Can a neural network learn to avoid wrong decisions using backpropagation?
I studied the articles on Neural Networks and Deep Learning from Michael Nielsen and developed a simple neural network based on his examples. I understand how backpropagation works and I already ...
4
votes
1answer
165 views
A few doubts on back propagation
I'm having trouble wrapping my head around some details of neural nets and back prop.
For example's sake, consider the following net, where I have separated the 'neurons' into linear nodes plus ...
4
votes
1answer
299 views
How to deal with back-propagation when dealing with invalid moves in Reinforced Learning?
As discussed in this thread, you can handle invalid moves in Reinforced Learning by re-setting the probabilities of all illegal moves to zero and renormalising the output vector.
In back-propagation, ...
3
votes
2answers
60 views
Could error surface shape be useful to detect which local minima is better for generalization?
The following plot shows error function output based on system weights.
Two equal local minima are shown in green pointers. Note that the red dots are not related to the question.
Does the right one ...
3
votes
1answer
168 views
Does training happen during NEAT?
When one uses NEAT to evolve the best fitting network for a task, does training take place in each epoch as well?
If I understand correctly, training is the adjustment of the weights of the neural ...
3
votes
1answer
23 views
Structure discrepancy of an LSTM?
I've found multiple depictions of how an LSTM cell operates. See 2 below:
and
Each of these images suggest the hidden state is utilised differently. On the top diagram, it is shown that the hidden ...
3
votes
1answer
56 views
How is REINFORCE used instead of Backpropagation?
In neural networks with stochastic layers I've seen the use of the REINFORCE estimator for estimating the gradient (because it can't be computed directly).
Some such examples are Show, Attend and ...
3
votes
1answer
45 views
What weights should I use while back-propagating?
I've started to learn about neural networks recently and I can't find the answer to this question.
Let's assume there's a neural network (fig. 1)
So if the loss function is:
and the derivative is:
...
3
votes
3answers
75 views
How does adding a small change to an neuron's weighted input affect the overall cost?
I was reading the following book: http://neuralnetworksanddeeplearning.com/chap2.html
and towards the end of equation 29, there is a paragraph that explains this:
However I am unsure how the ...
3
votes
1answer
65 views
Backpropagation equation for a variant on the usual Linear Neuron architecture
Recently I encountered a variant on the normal linear neural layer architecture: Instead of $Z = XW + B$, we now have $Z = (X-A)W + B$. So we have a 'pre-bias' $A$ that affects the activation of the ...
3
votes
0answers
488 views
How to calculate gradient of filter in convolution network
I have similar architecture like in image:CNN.
I don't understand how to calculate gradient of filter F.
I found these equations(source):
Gradient and delta,
where first equation calculate gradient ...
2
votes
2answers
3k views
How to combine backpropagation in neural nets and reinforcement learning?
As I am trying to make an AI with reinforcement learning, I have found out and implemented a lot of things such as both these topics (NNs and RL) separately. But when trying to combine them, I have ...
2
votes
3answers
2k views
What is the best XOR neural network configuration out there in terms of low error?
I'm trying to understand what would be the best neural network for implementing a XOR gate. I'm considering a neural network to be good if it can produce all the expected outcomes with the lowest ...
2
votes
1answer
227 views
Hand computing feed forward and back propagation of neural network
I used to treat back propagation as a black box but lately I want to understand more about it. I have used mattmuzr's and DuttA's explanaiton as a guide to hand compute a simple neural network. I have ...
2
votes
1answer
36 views
How is a neural network where the majority of inputs are 0 trained?
Take alexnet. Alexnet has 1000 output nodes, each of which classify an image:
The problem I have been having with training a neural network of similar proportions, is that it does what any reasonable ...
2
votes
1answer
89 views
What is the difference between backpropagation and predictive coding?
Reading the high-level descriptions of backpropagation and predictive coding, they don't sound so drastically different. What is the key difference between these techniques?
I am currently reading ...
2
votes
1answer
45 views
What is the use of the $\epsilon$ term in this back-propagation equation?
I am currently looking at different documents to understand back-propagation, mainly at this document. Now, at page 3, there is the $\epsilon$ symbol involved:
While I understand the main part of the ...
2
votes
1answer
38 views
When and how to use a mix of loss functions for back-propagation?
I am trying to understand the best loss function to be used with a convolutional neural network. I came to know that we can mix two loss functions. Can any body share in what case was it done and how?
2
votes
2answers
76 views
Which neuron represents which part of the input?
In a neural network, each neuron represents some part of the input. For example, in the case of a MNIST digit, consider the stem of the number 9. Each neuron in the NN represents some part of this ...
2
votes
0answers
95 views
Update of weights in Recurrent Neural Network through back propagation
How does Recurrent Neural Network updates its weights and bias through backpropagation?
Is time taken into account while updating the weights of a RNN using Backpropagation through time(BPTT)?"
2
votes
2answers
409 views
How to perform neural network with output constraint?
Imagine a "simple" feedforward, fully connected neural network, with some input size, some number of hidden layers, and some # of neurons....etc BUT with a fixed number of output size (that is saying, ...
2
votes
1answer
102 views
How to determine the size of biases? [closed]
I'm new to the world of machine learning. My question is how can I determine the size of the biases in a neural network (with backpropagation algorithm)? Currently, I have a 2-layer neural network (1 ...
