Questions tagged [backpropagation]

For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.

Filter by
Sorted by
Tagged with
10 votes
2 answers
5k views

What exactly is averaged when doing batch gradient descent?

I have a question about how the averaging works when doing mini-batch gradient descent. I think I now understood the general gradient descent algorithm, but only for online learning. When doing mini-...
Ben's user avatar
  • 435
41 votes
4 answers
56k views

What is the time complexity for training a neural network using back-propagation?

Suppose that a NN contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_i$ nodes in each layer. What is the time complexity to train this NN using back-propagation? I have a basic ...
user avatar
9 votes
1 answer
10k views

Is back-propagation applied for each data point or for a batch of data points?

I am new to deep learning and trying to understand the concept of back-propagation. I have a doubt about when the back-propagation is applied. Assume that I have a training data set of 1000 images ...
Maanu's user avatar
  • 235
5 votes
3 answers
2k views

What is the actual learning algorithm: back-propagation or gradient descent?

What is the actual learning algorithm: back-propagation or gradient descent (or, in general, the optimization algorithm)? I am reading through chapter 8 of Parallel Distributed Processing hand book ...
Sreedhar Veluri's user avatar
10 votes
2 answers
771 views

What are the learning limitations of neural networks trained with backpropagation?

In 1969, Seymour Papert and Marvin Minsky showed that Perceptrons could not learn the XOR function. This was solved by the backpropagation network with at least one hidden layer. This type of network ...
S.L. Barth is on codidact.com's user avatar
7 votes
1 answer
2k views

Why do we update all layers simultaneously while training a neural network?

Very deep models involve the composition of several functions or layers. The gradient tells how to update each parameter, under the assumption that the other layers do not change. In practice, we ...
stoic-santiago's user avatar
6 votes
1 answer
945 views

Does training happen during NEAT?

When one uses NEAT to evolve the best fitting network for a task, does training take place in each epoch as well? If I understand correctly, training is the adjustment of the weights of the neural ...
Alexus's user avatar
  • 236
5 votes
2 answers
1k views

Are filters fixed or learned?

No matter what I google or what paper I read, I can't find an answer to my question. In a deep convolutional neural network, let's say AlexNet (Krizhevsky, 2012), filters' weights are learned by means ...
nvergontbij's user avatar
4 votes
1 answer
1k views

Why is second-order backpropagation useful?

Raul Rojas's book on Neural Networks dedicates section 8.4.3 to explaining how to do second-order backpropagation, that is, computing the Hessian of the error function with respect to two weights at a ...
EmmanuelMess's user avatar
8 votes
3 answers
2k views

How does backprop work through the random sampling layer in a variational autoencoder?

Implementations of variational autoencoders that I've looked at all include a sampling layer as the last layer of the encoder block. The encoder learns to generate a mean and standard deviation for ...
Luke Wolcott's user avatar
8 votes
1 answer
1k views

Which loss function should I use in REINFORCE, and what are the labels?

I understand that this is the update for the parameters of a policy in REINFORCE: $$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}, $$ where $v_t$ is ...
S2673's user avatar
  • 580
6 votes
1 answer
5k views

How to avoid falling into the "local minima" trap?

How do I avoid my gradient descent algorithm into falling into the "local minima" trap while backpropogating on my neural network? Are there any methods which help me avoid it?
Dawny33's user avatar
  • 1,371
6 votes
2 answers
2k views

In deep learning, is it possible to use discontinuous activation functions?

In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)? (My guess: for example, ReLU is non-differentiable at a single point, but it still has a ...
Gyeonghoon Ko's user avatar
5 votes
1 answer
568 views

How does backpropagation work on a custom loss function whose components have magnitudes of different orders?

I want to use a custom loss function which is a weighted combination of l1 and DSSIM losses. The DSSIM loss is limited between 0 and 0.5 where as the l1 loss can be orders of magnitude greater and is ...
user12754's user avatar
  • 107
4 votes
2 answers
7k views

How do I calculate the gradient of the hinge loss function?

With reference to the research paper entitled Sentiment Embeddings with Applications to Sentiment Analysis, I am trying to implement its sentiment ranking model in Python, for which I am required to ...
Raj Shrivastava's user avatar
4 votes
1 answer
297 views

Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?

I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation. As per my knowledge, ...
learner's user avatar
  • 151
3 votes
1 answer
483 views

Are my computations of the forward and backward pass of a neural network with one input, hidden and output neurons correct?

I have computed the forward and backward passes of the following simple neural network, with one input, hidden, and output neurons. Here are my computations of the forward pass. \begin{align} net_1 &...
Eka's user avatar
  • 1,066
2 votes
1 answer
64 views

How to improve a trained model over time (i.e. with more predictions)?

I built a model using the tutorial on the TensorFlow site. It was a simple image classification neural network. I trained it and saved the model and weights together on a ...
Rutvik Karupothula's user avatar
1 vote
1 answer
282 views

Do you need to store prevous values of weights and layers on recurrent layer while BPTT?

The Back propagation through time on recurrent layer is defined similar to normal one, means somethin like ...
user8426627's user avatar
1 vote
2 answers
2k views

How should I implement the backward pass through a flatten layer of a CNN?

I am making a NN library without any other external NN library, so I am implementing all layers, including the flatten layer, and algorithms (forward and backward pass) from scratch. I know the ...
Clement's user avatar
  • 1,735
0 votes
2 answers
270 views

How Does Convolution Backpropagation Work?

Assume in a convolutional layer's forward pass we have a $10\times10\times3$ image and five $3\times3\times3$ kernels, then $(10\times10\times3) *( 3\times3\times3\times5)$ has the output of ...
rkuang25's user avatar
-1 votes
1 answer
211 views

Why is the cross-entropy a cost function?

The question looks foolish, but I think cross-entropy is somewhat weird as a cost function. As a cost function for linear regression, the mean square error $ \sum_{i=1}^{n} (y_i - (ax_i+b)) ^2$ seems ...
JAEMTO's user avatar
  • 125