Questions tagged [backpropagation]

For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.

Filter by
Sorted by
Tagged with
-1 votes
0 answers
20 views

CNN image classification [closed]

While training CNN with a fully connected layer for image classification, isn't training everything at once the problem? For example, we want to classify dogs. Somehow in the first epoch feature ...
user avatar
  • 1
0 votes
0 answers
46 views

What method to use when optimizing an array of data

Say I have an array of data, where each element describes a shape made of points, in vector form (each vector has several hundred dimensions). Each element also has a rating that gets higher, the ...
user avatar
1 vote
1 answer
34 views

Trouble writing the backpropagation algorithm in python through crossentropy and softmax

so I am writing my own neural network library for a class project and I got everything working for a simple 2-class test using the distance (L2) cost function. I wanted to get a similar result using ...
user avatar
1 vote
1 answer
44 views

Is there some kind of "weighted maximum" that allows the gradients to backpropagate? [closed]

I was wanting to add a maximum in my neural network, but this seems a bad thing to do since it kills the gradients to all but one of the inputs. Is there some kind of "weighted maximum" that ...
user avatar
0 votes
0 answers
15 views

Recursive Memory Optimized Gradient Graph Explained?

I'm reading the paper Training Deep Nets with Sublinear Memory Cost by Tianqi Chen, et. al. The paper is known for the $O(\sqrt n)$ memory cost to train a $n$-layer neural network. My problem is ...
user avatar
  • 111
8 votes
1 answer
1k views

What do symmetric weights mean and how does it make backpropagation biologically implausible?

I was reading a paper on alternatives to backpropagation as a learning algorithm in neural networks. In this paper, the author talks about the disadvantages of backpropagation, and one of the ...
user avatar
  • 83
1 vote
0 answers
17 views

In Graph Neural Network is Message Passing Step Agnostic of Output Values during Training?

So Graph Neural Networks is about representation learning where initially representation of graph is learned in the form of node embeddings. My question is: Are the output values back propagated and ...
user avatar
  • 135
0 votes
0 answers
10 views

Can someone give me an example that shows the working of Vector mode Forward Automatic Differentiation?

Given a Function F(x,y,z), and I want to calculate the derivative of the function with respect to x,y and z, forward mode generally will take 3 passes to compute the derivative, one pass for each x,y ...
user avatar
1 vote
0 answers
41 views

Do we need backpropagation if there is only one class?

A am interested in physiologic neural network. Altough there are some opposite views, most probably there seems to be no plausible way to explain a physiologic backpropagation in the brain. So I am ...
user avatar
  • 11
0 votes
0 answers
21 views

Is the Adam optimizer moment values different for each layer?

If I have an 3 tensorflow layers in a network and the 2 weights between these layers are of different dimensions, how does the Adam optimizer algorithm work? For the pseudocode in https://optimization....
user avatar
0 votes
0 answers
25 views

Why do I have a shape mismatch when running back propagation of convolutional layer in CNN?

I am having problems with my array shapes when calculating change in loss wrt Kernel (delL/delk) in back propagation of convolutional layer. I am running a mini batch neural network operating on 32 ...
user avatar
  • 11
0 votes
0 answers
21 views

Is there a vectorized Implementation of Convolution (Full Mode) with a very big kernel/gradient?

I'm currently trying to figure a way to implement the backpropagation of a convolutional layer with plain numpy. In theory, I can calculate the partial derivative of the loss w.r.t. the convolution ...
user avatar
1 vote
1 answer
56 views

Do neural networks, trained with backpropagation algorithm, exploit the concept of synaptic plasticity?

Is there some of Hebb's rule behind the concept of backpropagation learning rule of a simple supervised neural network, that for example is trained for classification task ? I was reading about the ...
user avatar
0 votes
0 answers
26 views

How to calculate the gradient (or derivative) of y = f(x) of y w.r.t x where y represents the order statistics divided by median of x?

How to calculate the gradient (or derivative) of y = f(x) of y w.r.t x where y represents the order statistics divided by median of x? For instance x is ...
user avatar
  • 101
0 votes
0 answers
41 views

Is my calculation of the partial derivative of the cost function with respect to a single weight in the first layer correct?

I'm trying to understand the chain rule of backpropagation. This is what I understood. Is it correct? $$ \frac{\partial E }{ \partial w} = \sum_{i} \frac{\partial E }{ \partial a_i^{(l)} } (\sum_{j} \...
user avatar
-1 votes
1 answer
89 views

Why is the cross-entropy a cost function?

The question looks foolish, but I think cross-entropy is somewhat weird as a cost function. As a cost function for linear regression, the mean square error $ \sum_{i=1}^{n} (y_i - (ax_i+b)) ^2$ seems ...
user avatar
  • 125
0 votes
1 answer
27 views

Deep Learning Architecture where outputs from two different inputs are used for error calculation

Is there a deep learning architecture where outputs of the same model with two different inputs are used for error calculation (backpropagation)? Workflow: Input1 -----> Model ------> Output1 ...
user avatar
6 votes
3 answers
241 views

How does backprop work through the random sampling layer in a variational autoencoder?

Implementations of variational autoencoders that I've looked at all include a sampling layer as the last layer of the encoder block. The encoder learns to generate a mean and standard deviation for ...
user avatar
0 votes
1 answer
66 views

Why "large set of training data" is needed in Neural Network AI training?

I often heard people saying, "large set of training data is needed for producing an accurate AI". But when I looked for articles explaining backpropagations online, it all seems like you ...
user avatar
  • 11
3 votes
0 answers
46 views

How are partial derivatives calculated in a computational graph?

I am trying to understand how are partial derivatives calculated in a computational graph. I understand reasoning behind computational graphs and I am bold enough to say I understand how they work, at ...
user avatar
  • 131
1 vote
0 answers
28 views

CNN: Difficulties understanding backward pass derivatives

I have really quite hard difficulties to understand what is actually going on in the backward pass of a CNN. I am currently focusing on these references: https://towardsdatascience.com/forward-and-...
user avatar
0 votes
0 answers
51 views

In mini-batch gradient descent, are the weights updated after each batch or after all the batches have gone through an epoch?

Say I have a mini-batch of size 32, and I have 10 such batches. Assuming I only run it for one epoch (just for the sake of understanding it), Will the weights be updated using the gradients of one ...
user avatar
1 vote
1 answer
61 views

How does backpropagation know which weights to change?

I'm currently working on constructing a neural network from scratch (in JavaScript). I'm in the middle of working on the backpropagation, but there's something I don't understand: how does the ...
user avatar
0 votes
1 answer
37 views

Does the ANN's training data include the proper output for every neuron?

I was designing an Artificial Neural Network a while back, but hit a bump when I got to the backpropagation. I was having trouble making the script choose whether to add or subtract from the weights, ...
user avatar
0 votes
0 answers
19 views

How Long Can BPTT Truncated?

I wanted to ask what is, in general, the maximum value (the order of magnitude) of the number of time steps I can back-propagate in the past using TBTT (Truncated Backpropagation Through Time) in an ...
user avatar
0 votes
0 answers
21 views

Discrepancy of backpropagation formula between Andrew Ngs ML Course and those derived by neuralnetworksanddeeplearning.com

I'm currently working through Week 5 of Andrew Ngs Machine Learning course on Coursera, which goes through the backprop algorithm for basic neural networks. Whilst trying to derive the formulae he ...
user avatar
0 votes
0 answers
33 views

RNN - Backpropagation through time - Gradient Calculation

I think I got it right after reading multiple resources but im still not 100%. Seems like everyone is calculating it different. Or they just shortcut explaining the calculation. (or my math skills ...
user avatar
1 vote
1 answer
148 views

Different ways to calculate backpropagation derivatives, any difference?

I'm studying error backpropagation in neural networks. I am interested in why we use only one path on the computational graph to get the value of the derivative for a weight? I ask the question ...
user avatar
1 vote
1 answer
35 views

Parallelize Backpropagation - How to synchronize the weights of each thread?

I implemented a parallel backpropagation algorithm that uses $n$ threads. Now every thread gets $\dfrac{1}{n}$ examples of the training data and updates its instance of the net with it. After every ...
user avatar
  • 11
2 votes
0 answers
79 views

How does back propagation adjust the hidden layers' weights and biases?

I'm new to neural networks and trying to figure out its fundamentals but I cannot fully understand the back propagation algorithm. In back propagation, I understand we want to go backwards from the ...
user avatar
1 vote
0 answers
31 views

ReLU function converging to local optimum in one case and diverging in the other one

I implemented a simple neural network with 1 hidden layer. I used ReLU as activation function for the hidden layer and the output layer just uses the linear function. To check my implementation I ...
user avatar
2 votes
1 answer
80 views

How to create a neural network from a set of equations?

Say I have these equations: $$x_1 = x_2 + 2y_1 + b$$ $$x_2 = y_2 + c$$ $$y_1 = z + a$$ $$y_2 = y_3 + d$$ $$z = z_1 + e$$ $x_1$ depends on $x_2$ (depends on $y_2$ (depends on $y_3$)) and $y_1$ (depends ...
user avatar
  • 121
1 vote
0 answers
52 views

Why doesn't anyone use reinforcement learning to find the best possible alternative to backpropagation?

To be clear, I'm very uninformed on the topic of alternative learning algorithms to backprop, all my knowledge comes from articles like these: lets-not-stop-at-backprop backprop-alternatives we-need-a-...
user avatar
  • 103
0 votes
0 answers
240 views

What are the input and output gradients in PyTorch?

Suppose I want to train a neural network with $m-$length inputs of form $x = [x_1, x_2, x_3, \cdots, x_m]$ and $n-$length outputs of form $y = [y_1, y_2, y_3, \cdots, y_n]$. Let the number of ...
user avatar
  • 3,099
0 votes
0 answers
24 views

when updating the bias matrix, do we get the total sum of dZ or the sum of the axis of dZ?

I'm currently studying how to implement a neural network from scratch to know how it works, I came across this article: https://www.samsonzhang.com/2020/11/24/understanding-the-math-behind-neural-...
user avatar
2 votes
1 answer
63 views

What does "differentiable architecture" mean?

I'm currently reading a paper that uses CNN's as a base approach to solving some image classification issues and I've found that they kept mentioning the term "Differentiable Architecture", ...
user avatar
3 votes
1 answer
152 views

Can some of the weights be fixed during the training of a neural network?

Is it possible to exclude specific layers from the optimization? For example, let's say I have an input layer, 2 hidden layers, and the output layer. I know there is a perfect solution for my problem ...
user avatar
1 vote
0 answers
51 views

Backpropagation not working as expected

I'm new to neural networks and I try to make a model that is guessing if a point is below or above relative to a function output. The idea is inspired from this video https://youtu.be/DGxIcDjPzac . ...
user avatar
  • 111
0 votes
0 answers
26 views

Are there any good references that describe the equations of the forward pass of Graph Neural Networks?

I am trying to program Graph Neural Network from scratch. Can the community please suggest a good reference/s to read about the equations of the forward pass in Graph Neural Networks, especially in ...
user avatar
  • 121
1 vote
0 answers
203 views

Convolutional Layer Multichannel Backpropagation Implementation

I have been working on coding a CNN in python from scratch using numpy as a semester project and I think I have successfully implemented it up to backpropagation in the MaxPool Layers. However, my ...
user avatar
0 votes
1 answer
137 views

What is the correct formula for updating the weights in a 1-single hidden layer neural network?

I'm creating a neural network with 3 layers and no bias. On internet I saw that the expression for the derivative of the weights between the hidden layer and the output layer was: $$\Delta W_{j,k} = (...
user avatar
0 votes
0 answers
29 views

Backpropagation - what does rate of change calculated from the partial derivatives actually relate to?

I understand conceptually how backpropagation works according to the chain rule, and I understand that partial derivatives calculate the rate of change of a function containing multiple variables with ...
user avatar
3 votes
1 answer
296 views

Is the bias also a "weight" in a neural network?

I'm learning about how neural networks are trained. I understand how a neuron works, backpropagation, and all that. In neurons, there is a clear distinction between a "weight" and a "...
user avatar
2 votes
1 answer
232 views

Why is the backpropagation algorithm used to train the multilayer perceptron?

I've read in the book Neural Network Design, by Martin Hagan et al. (chapter 11), that, to train the feed-forward neural network (aka multilayer perceptron), one uses the backpropagation algorithm. ...
user avatar
2 votes
1 answer
58 views

How to improve a trained model over time (i.e. with more predictions)?

I built a model using the tutorial on the TensorFlow site. It was a simple image classification neural network. I trained it and saved the model and weights together on a ...
user avatar
1 vote
0 answers
27 views

In a convolutional neural network, how is the error delta propagated between convolutional layers?

I'm coding some stuff for CNNs, just relying on numpy (and scipy just for the convolution operation for pure performance reasons). I've coded a small network consisting of a convolutional layer with ...
user avatar
  • 175
0 votes
1 answer
47 views

Are there relatively new research papers that describe how to make back-propagation more efficient?

I read Yann LeCun's paper Efficient BackProp, which was published in 2000. I looked for similar but more recent papers on Arxiv, but I have not yet found any. Are there relatively new research papers ...
user avatar
  • 101
4 votes
1 answer
273 views

Why is it a problem if the outputs of an activation function are not zero-centered?

In this lecture, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that ...
user avatar
  • 509
0 votes
0 answers
34 views

Is vectorizing backpropagation feasible?

Does it make sense to have the backpropagation of a neural network layer happen all at once if the learning rate is lowered? This would mean the new weights of that layer would be independent of each ...
user avatar
  • 126
1 vote
1 answer
83 views

In gradient descent's update rule, why do we use $\sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}$ instead of $\frac{\delta C_0}{\delta w^{l}}$?

I am trying to code a two layered neural network simple NN as I have described here https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/ I am getting stuck on the last step of updating the ...
user avatar

1
2 3 4 5