# Questions tagged [backpropagation]

For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.

226 questions
Filter by
Sorted by
Tagged with
20 views

### CNN image classification [closed]

While training CNN with a fully connected layer for image classification, isn't training everything at once the problem? For example, we want to classify dogs. Somehow in the first epoch feature ...
46 views

### What method to use when optimizing an array of data

Say I have an array of data, where each element describes a shape made of points, in vector form (each vector has several hundred dimensions). Each element also has a rating that gets higher, the ...
1 vote
34 views

### Trouble writing the backpropagation algorithm in python through crossentropy and softmax

so I am writing my own neural network library for a class project and I got everything working for a simple 2-class test using the distance (L2) cost function. I wanted to get a similar result using ...
1 vote
44 views

### Is there some kind of "weighted maximum" that allows the gradients to backpropagate? [closed]

I was wanting to add a maximum in my neural network, but this seems a bad thing to do since it kills the gradients to all but one of the inputs. Is there some kind of "weighted maximum" that ...
15 views

### Recursive Memory Optimized Gradient Graph Explained?

I'm reading the paper Training Deep Nets with Sublinear Memory Cost by Tianqi Chen, et. al. The paper is known for the $O(\sqrt n)$ memory cost to train a $n$-layer neural network. My problem is ...
1k views

### What do symmetric weights mean and how does it make backpropagation biologically implausible?

I was reading a paper on alternatives to backpropagation as a learning algorithm in neural networks. In this paper, the author talks about the disadvantages of backpropagation, and one of the ...
1 vote
17 views

### In Graph Neural Network is Message Passing Step Agnostic of Output Values during Training?

So Graph Neural Networks is about representation learning where initially representation of graph is learned in the form of node embeddings. My question is: Are the output values back propagated and ...
10 views

### Can someone give me an example that shows the working of Vector mode Forward Automatic Differentiation?

Given a Function F(x,y,z), and I want to calculate the derivative of the function with respect to x,y and z, forward mode generally will take 3 passes to compute the derivative, one pass for each x,y ...
1 vote
41 views

### Do we need backpropagation if there is only one class?

A am interested in physiologic neural network. Altough there are some opposite views, most probably there seems to be no plausible way to explain a physiologic backpropagation in the brain. So I am ...
21 views

### Is the Adam optimizer moment values different for each layer?

If I have an 3 tensorflow layers in a network and the 2 weights between these layers are of different dimensions, how does the Adam optimizer algorithm work? For the pseudocode in https://optimization....
25 views

### Why do I have a shape mismatch when running back propagation of convolutional layer in CNN?

I am having problems with my array shapes when calculating change in loss wrt Kernel (delL/delk) in back propagation of convolutional layer. I am running a mini batch neural network operating on 32 ...
21 views

### Is there a vectorized Implementation of Convolution (Full Mode) with a very big kernel/gradient?

I'm currently trying to figure a way to implement the backpropagation of a convolutional layer with plain numpy. In theory, I can calculate the partial derivative of the loss w.r.t. the convolution ...
1 vote
56 views

### Do neural networks, trained with backpropagation algorithm, exploit the concept of synaptic plasticity?

Is there some of Hebb's rule behind the concept of backpropagation learning rule of a simple supervised neural network, that for example is trained for classification task ? I was reading about the ...
26 views

### How to calculate the gradient (or derivative) of y = f(x) of y w.r.t x where y represents the order statistics divided by median of x?

How to calculate the gradient (or derivative) of y = f(x) of y w.r.t x where y represents the order statistics divided by median of x? For instance x is ...
41 views

29 views

### Backpropagation - what does rate of change calculated from the partial derivatives actually relate to?

I understand conceptually how backpropagation works according to the chain rule, and I understand that partial derivatives calculate the rate of change of a function containing multiple variables with ...
296 views

### Is the bias also a "weight" in a neural network?

I'm learning about how neural networks are trained. I understand how a neuron works, backpropagation, and all that. In neurons, there is a clear distinction between a "weight" and a "...
232 views

### Why is the backpropagation algorithm used to train the multilayer perceptron?

I've read in the book Neural Network Design, by Martin Hagan et al. (chapter 11), that, to train the feed-forward neural network (aka multilayer perceptron), one uses the backpropagation algorithm. ...
58 views

### How to improve a trained model over time (i.e. with more predictions)?

I built a model using the tutorial on the TensorFlow site. It was a simple image classification neural network. I trained it and saved the model and weights together on a ...
1 vote
27 views

### In a convolutional neural network, how is the error delta propagated between convolutional layers?

I'm coding some stuff for CNNs, just relying on numpy (and scipy just for the convolution operation for pure performance reasons). I've coded a small network consisting of a convolutional layer with ...
47 views

### Are there relatively new research papers that describe how to make back-propagation more efficient?

I read Yann LeCun's paper Efficient BackProp, which was published in 2000. I looked for similar but more recent papers on Arxiv, but I have not yet found any. Are there relatively new research papers ...
273 views

### Why is it a problem if the outputs of an activation function are not zero-centered?

In this lecture, the professor says that one problem with the sigmoid function is that its outputs aren't zero-centered. Are the explanation provided by the professor regarding why this is bad is that ...