Questions tagged [backpropagation]
For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.
240
questions
0
votes
1
answer
48
views
Seq2seq with RNNs, how is the training loop performed?
How do we train a seq2seq rnn training?
We input a sentence that needs to be translated. We encode it sequentially. Then the first decoder outputs the first word with probabilities. We do a gradient ...
1
vote
3
answers
58
views
Do Artificial Neural Network with non-linear activation only in the output layer follows linearity?
I am using a model with linear activation in the hidden layer and non-linear activation in the output layer. Could you please help to know whether such models exhibit linearity or not?
The non-linear ...
0
votes
0
answers
8
views
Multiple network with one loss function, non-differentiable, is it okay with backpropagation?
Hello I'm trying to make the augmentation function which chooses to augmentate or not based on the contents of image.
To make this, I'm thinking about using two different network.
One is to classify ...
- 1
0
votes
0
answers
39
views
How does the cross entropy loss function interact with the final layer of a neural network?
I am having trouble understanding how the result of categorical cross entropy loss can be used to calculate the gradient for all of the weights.
The output of cross entropy function is the sum of all ...
- 1
2
votes
0
answers
45
views
Backpropagation with multiple output neurons but only one loss value
Suppose we have the following neural network (in reality it is a CNN with 60k parameters):
This image, as well as the terminology used here, is borrowed from Matt Mazur
As is visible, there are two ...
0
votes
0
answers
60
views
When should I think of using the Forward Forward algorithm?
Recently I read the paper named "The Forward-Forward Algorithm: Some Preliminary Investigations" and I was wondering what cases I should think of using it on.
So according to the paper If we ...
0
votes
1
answer
42
views
Single Layer Perceptron Backpropagation: How to compute affect of the net value on the output?
Assuming a single perceptron (see figure), I have found two versions of how to use backpropagation to update the weights. The perceptron is split in two, so we see the weighted sum on the left (the ...
- 1
0
votes
0
answers
27
views
Why differentiation of fourier operator is difficult?
I have a question when I read some papers about physics-informed neural networks.
In the paper of physics-informed neural operator, they said "it is non-trivial to compute the derivatives for ...
- 1
0
votes
1
answer
48
views
Is it possible to reverse engineer out the loss based on weights update when data is unknown?
Assume the gradient updates (both $W_t$ and $W_{t+1}$) and learning rate are known while data $X$ is unknown, is it possible to deduce the loss $L$ used in backprop algorithm that gave rise to the ...
- 165
2
votes
2
answers
276
views
Why should one expect the backward pass to take twice as long as the forward pass?
I have seen it stated that, as a rule of thumb, a backward pass in a neural network should take about twice as long as the forward pass. Examples:
From DeepSpeed's Flops Profiler docs, the profiler:
...
- 121
2
votes
2
answers
66
views
Back propagation activation function derivative [closed]
I am reading about backpropagation for fully connected neural networks and I found a very interesting article by Jeremy Jordan. It explains the process from start to finish. There is a section though ...
1
vote
0
answers
20
views
How to make my neural networks designs more robust
Whenever, I design a neural network to solve a novel problem (requires a novel loss function i.e. not image classification) it always ends up being hypersensitive to batch size and learning rate.
...
- 288
0
votes
2
answers
111
views
How Does Convolution Backpropagation Work?
Assume in a convolutional layer's forward pass we have a $10\times10\times3$ image and five $3\times3\times3$ kernels, then $(10\times10\times3) *( 3\times3\times3\times5)$ has the output of ...
- 21
1
vote
0
answers
42
views
Training loss is decreasing very slowly while learning MNIST database
I am developing my ANN from scratch which is supposed to classify MNIST database of handwritten digits (0-9). My feed-forward fully connected ANN has to be composed of:
One input layer, with ...
- 11
0
votes
0
answers
60
views
Very low accuracy (0.11) and it remains constant after few epochs on MNIST database
I am developing my ANN from scratch which is supposed to classify MNIST database of handwritten digits (0-9). My feed-forward fully connected ANN has to be composed of:
One input layer, with ...
- 137
0
votes
1
answer
33
views
Is my neural network working?
I recently just finished programming a neural network in c#, and it seems like it's working. My question is if I'm doing it right. It's a very confusing process so I will explain.
Basically every ...
- 3
0
votes
0
answers
63
views
How does backpropagation work with multi-branch models?
How does backpropagation work when the input layer feeds into two or more separate branches of layers before merging back to produce a single output, such as can be implemented in the Keras Functional ...
0
votes
0
answers
33
views
The result of back propagation for a neural network
I have created a neural network that feeds an image into a convolutional neural net, then feeds the flattened output of this network into an artificial neural network.
I have a feeling that my ...
- 1
0
votes
1
answer
41
views
Are there limitations on network output architecture and action mapping in reinforcement learning?
I'm easing my way into a toy reinforcement learning problem where my objects can move and take different actions on a simple grid, but I'm having trouble understanding what constraints might exist in ...
- 89
0
votes
1
answer
39
views
Why is the derivative of activation function all positive?
All the activation functions I see have positive derivatives.
Will negative ReLU work as well as its positive counterpart or will it lead to instability?
- 101
-1
votes
1
answer
148
views
In a neural network's neuron that has no activation function, to calculate the delta for the neuron during back propagation do you use a derivative?
I have a neural network that is composed of an input layer, two hidden layers and an output layer. The topology is [151, 200, 100, 1] I am using ReLU activation function on the neurons that are in the ...
- 1
0
votes
0
answers
16
views
Simple dimension unmatch problem of a simple neural network
In this simple neural network:
the derivative for the cost function J when assuming binary cross entropy loss would be
If we assume that the dimension of X is 2x1, then wouldn't A1 be 2x1 and A2 be ...
- 1
0
votes
0
answers
52
views
Why Is There The Term 1/m In Backpropagation
In backpropagation the gradients are used to update the weights using the formula
$$w = w - \alpha \frac{dL}{dw}$$
and the loss gradient w.r.t. weights is
$$\frac{dL}{dw} = \frac{dL}{dz} \frac{dz}{dw} ...
- 21
1
vote
1
answer
122
views
Why do we use gradient descent to minimize the loss function?
The purpose of training neural networks is to minimize a loss function, in this process we usually use gradient descent method.
But in Calculus, if we want to find the global minimum of a ...
- 111
0
votes
2
answers
130
views
Is this the correct way to backpropagate a Neural Network?
I am writing a Neural Network frorm scratch. Below is what I have right now, based off of the math that I think I understand.
...
- 15
0
votes
0
answers
79
views
What method to use when optimizing an array of data
Say I have an array of data, where each element describes a shape made of points, in vector form (each vector has several hundred dimensions). Each element also has a rating that gets higher, the ...
1
vote
1
answer
171
views
Trouble writing the backpropagation algorithm in python through crossentropy and softmax
so I am writing my own neural network library for a class project and I got everything working for a simple 2-class test using the distance (L2) cost function. I wanted to get a similar result using ...
- 121
1
vote
1
answer
57
views
Is there some kind of "weighted maximum" that allows the gradients to backpropagate? [closed]
I was wanting to add a maximum in my neural network, but this seems a bad thing to do since it kills the gradients to all but one of the inputs.
Is there some kind of "weighted maximum" that ...
- 288
0
votes
0
answers
18
views
Recursive Memory Optimized Gradient Graph Explained?
I'm reading the paper Training Deep Nets with Sublinear Memory Cost by Tianqi Chen, et. al. The paper is known for the $O(\sqrt n)$ memory cost to train a $n$-layer neural network.
My problem is ...
- 111
8
votes
1
answer
1k
views
What do symmetric weights mean and how does it make backpropagation biologically implausible?
I was reading a paper on alternatives to backpropagation as a learning algorithm in neural networks. In this paper, the author talks about the disadvantages of backpropagation, and one of the ...
- 83
1
vote
0
answers
23
views
In Graph Neural Network is Message Passing Step Agnostic of Output Values during Training?
So Graph Neural Networks is about representation learning where initially representation of graph is learned in the form of node embeddings. My question is: Are the output values back propagated and ...
- 145
0
votes
0
answers
16
views
Can someone give me an example that shows the working of Vector mode Forward Automatic Differentiation?
Given a Function F(x,y,z), and I want to calculate the derivative of the function with respect to x,y and z, forward mode generally will take 3 passes to compute the derivative, one pass for each x,y ...
1
vote
0
answers
45
views
Do we need backpropagation if there is only one class?
A am interested in physiologic neural network. Altough there are some opposite views, most probably there seems to be no plausible way to explain a physiologic backpropagation in the brain.
So I am ...
user53603
1
vote
1
answer
108
views
Do neural networks, trained with backpropagation algorithm, exploit the concept of synaptic plasticity?
Is there some of Hebb's rule behind the concept of backpropagation learning rule of a simple supervised neural network, that for example is trained for classification task ?
I was reading about the ...
0
votes
0
answers
119
views
Is my calculation of the partial derivative of the cost function with respect to a single weight in the first layer correct?
I'm trying to understand the chain rule of backpropagation.
This is what I understood. Is it correct?
$$ \frac{\partial E }{ \partial w} = \sum_{i} \frac{\partial E }{ \partial a_i^{(l)} } (\sum_{j} \...
-1
votes
1
answer
159
views
Why is the cross-entropy a cost function?
The question looks foolish, but I think cross-entropy is somewhat weird as a cost function.
As a cost function for linear regression, the mean square error $ \sum_{i=1}^{n} (y_i - (ax_i+b)) ^2$ seems ...
- 125
0
votes
1
answer
30
views
Deep Learning Architecture where outputs from two different inputs are used for error calculation
Is there a deep learning architecture where outputs of the same model with two different inputs are used for error calculation (backpropagation)?
Workflow:
Input1 -----> Model ------> Output1
...
8
votes
3
answers
811
views
How does backprop work through the random sampling layer in a variational autoencoder?
Implementations of variational autoencoders that I've looked at all include a sampling layer as the last layer of the encoder block. The encoder learns to generate a mean and standard deviation for ...
- 183
0
votes
1
answer
72
views
Why "large set of training data" is needed in Neural Network AI training?
I often heard people saying, "large set of training data is needed for producing an accurate AI".
But when I looked for articles explaining backpropagations online, it all seems like you ...
- 11
3
votes
0
answers
67
views
How are partial derivatives calculated in a computational graph?
I am trying to understand how are partial derivatives calculated in a computational graph. I understand reasoning behind computational graphs and I am bold enough to say I understand how they work, at ...
- 131
1
vote
0
answers
50
views
CNN: Difficulties understanding backward pass derivatives
I have really quite hard difficulties to understand what is actually going on in the backward pass of a CNN.
I am currently focusing on these references:
https://towardsdatascience.com/forward-and-...
- 111
0
votes
1
answer
336
views
In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?
Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one ...
2
votes
1
answer
138
views
How does backpropagation know which weights to change?
I'm currently working on constructing a neural network from scratch (in JavaScript). I'm in the middle of working on the backpropagation, but there's something I don't understand: how does the ...
- 137
0
votes
1
answer
37
views
Does the ANN's training data include the proper output for every neuron?
I was designing an Artificial Neural Network a while back, but hit a bump when I got to the backpropagation. I was having trouble making the script choose whether to add or subtract from the weights, ...
- 137
0
votes
1
answer
45
views
Discrepancy of backpropagation formula between Andrew Ngs ML Course and those derived by neuralnetworksanddeeplearning.com
I'm currently working through Week 5 of Andrew Ngs Machine Learning course on Coursera, which goes through the backprop algorithm for basic neural networks. Whilst trying to derive the formulae he ...
1
vote
1
answer
403
views
Different ways to calculate backpropagation derivatives, any difference?
I'm studying error backpropagation in neural networks. I am interested in why we use only one path on the computational graph to get the value of the derivative for a weight? I ask the question ...
- 13
1
vote
1
answer
70
views
Parallelize Backpropagation - How to synchronize the weights of each thread?
I implemented a parallel backpropagation algorithm that uses $n$ threads. Now every thread gets $\dfrac{1}{n}$ examples of the training data and updates its instance of the net with it. After every ...
- 11
2
votes
0
answers
343
views
How does back propagation adjust the hidden layers' weights and biases?
I'm new to neural networks and trying to figure out its fundamentals but I cannot fully understand the back propagation algorithm.
In back propagation, I understand we want to go backwards from the ...
- 21
1
vote
0
answers
50
views
ReLU function converging to local optimum in one case and diverging in the other one
I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function.
To check my implementation I ...
- 147
2
votes
1
answer
92
views
How to create a neural network from a set of equations?
Say I have these equations:
$$x_1 = x_2 + 2y_1 + b$$
$$x_2 = y_2 + c$$
$$y_1 = z + a$$
$$y_2 = y_3 + d$$
$$z = z_1 + e$$
$x_1$ depends on $x_2$ (depends on $y_2$ (depends on $y_3$)) and $y_1$ (depends ...
- 121