Questions tagged [backpropagation]
For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.
22
questions
11
votes
2
answers
6k
views
What exactly is averaged when doing batch gradient descent?
I have a question about how the averaging works when doing mini-batch gradient descent.
I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-...
- 445
43
votes
4
answers
60k
views
What is the time complexity for training a neural network using back-propagation?
Suppose that a NN contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_i$ nodes in each layer. What is the time complexity to train this NN using back-propagation?
I have a basic ...
user9947
10
votes
1
answer
12k
views
Is back-propagation applied for each data point or for a batch of data points?
I am new to deep learning and trying to understand the concept of back-propagation. I have a doubt about when the back-propagation is applied. Assume that I have a training data set of 1000 images ...
- 245
5
votes
3
answers
2k
views
What is the actual learning algorithm: back-propagation or gradient descent?
What is the actual learning algorithm: back-propagation or gradient descent (or, in general, the optimization algorithm)?
I am reading through chapter 8 of Parallel Distributed Processing hand book ...
- 100
10
votes
2
answers
791
views
What are the learning limitations of neural networks trained with backpropagation?
In 1969, Seymour Papert and Marvin Minsky showed that Perceptrons could not learn the XOR function.
This was solved by the backpropagation network with at least one hidden layer. This type of network ...
7
votes
1
answer
2k
views
Why do we update all layers simultaneously while training a neural network?
Very deep models involve the composition of several functions or layers. The gradient tells how to update each parameter, under the assumption that the other layers do not change. In practice, we ...
- 1,161
6
votes
1
answer
1k
views
Does training happen during NEAT?
When one uses NEAT to evolve the best fitting network for a task, does training take place in each epoch as well?
If I understand correctly, training is the adjustment of the weights of the neural ...
- 236
5
votes
2
answers
1k
views
Are filters fixed or learned?
No matter what I google or what paper I read, I can't find an answer to my question. In a deep convolutional neural network, let's say AlexNet (Krizhevsky, 2012), filters' weights are learned by means ...
- 63
5
votes
1
answer
1k
views
Why is second-order backpropagation useful?
Raul Rojas's book on Neural Networks dedicates section 8.4.3 to explaining how to do second-order backpropagation, that is, computing the Hessian of the error function with respect to two weights at a ...
- 227
3
votes
1
answer
491
views
Are my computations of the forward and backward pass of a neural network with one input, hidden and output neurons correct?
I have computed the forward and backward passes of the following simple neural network, with one input, hidden, and output neurons.
Here are my computations of the forward pass.
\begin{align}
net_1 &...
- 1,076
8
votes
1
answer
1k
views
Which loss function should I use in REINFORCE, and what are the labels?
I understand that this is the update for the parameters of a policy in REINFORCE:
$$
\Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t},
$$
where $v_t$ is ...
- 590
8
votes
3
answers
2k
views
How does backprop work through the random sampling layer in a variational autoencoder?
Implementations of variational autoencoders that I've looked at all include a sampling layer as the last layer of the encoder block. The encoder learns to generate a mean and standard deviation for ...
- 183
7
votes
1
answer
5k
views
How to avoid falling into the "local minima" trap?
How do I avoid my gradient descent algorithm into falling into the "local minima" trap while backpropogating on my neural network?
Are there any methods which help me avoid it?
- 1,381
6
votes
2
answers
2k
views
In deep learning, is it possible to use discontinuous activation functions?
In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)?
(My guess: for example, ReLU is non-differentiable at a single point, but it still has a ...
5
votes
2
answers
7k
views
How do I calculate the gradient of the hinge loss function?
With reference to the research paper entitled Sentiment Embeddings with Applications to Sentiment Analysis, I am trying to implement its sentiment ranking model in Python, for which I am required to ...
5
votes
1
answer
597
views
How does backpropagation work on a custom loss function whose components have magnitudes of different orders?
I want to use a custom loss function which is a weighted combination of l1 and DSSIM losses. The DSSIM loss is limited between 0 and 0.5 where as the l1 loss can be orders of magnitude greater and is ...
- 107
4
votes
1
answer
318
views
Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?
I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation.
As per my knowledge, ...
- 151
2
votes
1
answer
66
views
How to improve a trained model over time (i.e. with more predictions)?
I built a model using the tutorial on the TensorFlow site. It was a simple image classification neural network. I trained it and saved the model and weights together on a ...
1
vote
1
answer
295
views
Do you need to store prevous values of weights and layers on recurrent layer while BPTT?
The Back propagation through time on recurrent layer is defined similar to normal one, means somethin like
...
- 358
1
vote
2
answers
2k
views
How should I implement the backward pass through a flatten layer of a CNN?
I am making a NN library without any other external NN library, so I am implementing all layers, including the flatten layer, and algorithms (forward and backward pass) from scratch. I know the ...
- 1,745
0
votes
2
answers
416
views
How Does Convolution Backpropagation Work?
Assume in a convolutional layer's forward pass we have a $10\times10\times3$ image and five $3\times3\times3$ kernels, then $(10\times10\times3) *( 3\times3\times3\times5)$ has the output of ...
- 31
-1
votes
1
answer
240
views
Why is the cross-entropy a cost function?
The question looks foolish, but I think cross-entropy is somewhat weird as a cost function.
As a cost function for linear regression, the mean square error $ \sum_{i=1}^{n} (y_i - (ax_i+b)) ^2$ seems ...
- 125