Questions tagged [backpropagation]

For questions related to the technique of backpropagation, whereby the loss, error, or correction signal calculated at the output of an artificial network output is fed back to the parameters in each layer of the network until the network's behavior converges to a training state within the required accuracy and reliability.

Filter by
Sorted by
Tagged with
1
vote
0answers
20 views

Doing backpropagation in an Tensorflow.js Neural Network

I have a neural network (which I am making from scratch). In order to make the neural network "learn" I need to conduct back-propagation. Using the code at the below how would I conduct back-...
0
votes
2answers
255 views

What is the proof behind the gradient of a curve being proportional to the distance between the two co-ordinates in the x-axis?

In the [delta rule][1] the equation to adjust the weight with respect to error is $$w_{(n+1)}=w_{(n)}-\alpha \times \frac{\partial E}{\partial w}$$ *where $\alpha$ is the learning rate and $E$ is the ...
2
votes
0answers
35 views

Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
4
votes
1answer
92 views

Backpropagation equation for a variant on the usual Linear Neuron architecture

Recently I encountered a variant on the normal linear neural layer architecture: Instead of $Z = XW + B$, we now have $Z = (X-A)W + B$. So we have a 'pre-bias' $A$ that affects the activation of the ...
0
votes
0answers
23 views

How to find the derivative of a dynamic neuron model, which depends on previous states of the neuron?

This is the equation where n denotes the current state, (n-1) denotes the state in the previous step etc. And to do back-propagation I need to find partial derivatives over each of the variables. For ...
0
votes
0answers
20 views

Finding the energy function given update rule of a single layer non-linear neural network

Consider the network with N neurons, each of which takes a $2 \times k$ input specified by the tuple $(\vec c_t, \vec \theta_t)$ to produce output $\vec{R}_t$ through an update rule on the pairwise ...
1
vote
3answers
100 views

Why does a neuron in a multi-layer network need several input connections?

For example, if I have the following architecture: Each neuron in the hidden layer has a connection from each one in the input layer. 3 x 1 Input Matrix and a 4 x 3 weight matrix (for the ...
6
votes
2answers
83 views

Why do very deep non resnet architectures perform worse compared to shallower ones for the same iteration? Shouldn't they just train slower?

My understanding of the vanishing gradient problem in deep networks is that as backprop progresses through the layers the gradients become small, and thus training progresses slower. I'm having a hard ...
0
votes
0answers
55 views

Backpropagation implementation with Java

I've been trying to implement a Multilayer Perceptron Network using java language with the ultimate goal of creating and teaching a neural network to recognize handwritten digits. Pretty simple and ...
4
votes
0answers
70 views

What should I do with the flatten layer during back-propagation?

I'm creating a CNN network without other frameworks such as PyTorch, Keras, Tensorflow, and so on. During the forward pass, the Flatten layer reshapes the previous ...
0
votes
1answer
22 views

Positive bias causes the calculation of incorrect gradients

I have a data set with a positive bias (an image, where the values range from 0 to 1), that seems to be causing my network to calculate incorrect gradients. If I just use the raw image as input, of ...
0
votes
0answers
41 views

Why is it that having a duplicate in features set makes training to work bad

I'm defining a deep network to emulate a multitarget regression. When I costruct my training set, I take information from a graph; without going into too much detail, it could happen that I take 2 ...
0
votes
1answer
59 views

Is my understanding of back-propogation correct?

I am trying to learn backpropagation and this is what I know so far. To update the weights of the neural network you have to figure out the partial derivative of each of the parameters on the loss ...
1
vote
0answers
56 views

I need help understanding general back propagation algorithm

In section 6.5.6 of the book Deep Learning by Ian et. al. general backpropagation algorithm is described as: The back-propagation algorithm is very simple. To compute the gradient of some scalar z ...
3
votes
2answers
969 views

How to perform neural network with output constraint?

Imagine a "simple" feedforward, fully connected neural network, with some input size, some number of hidden layers, and some # of neurons....etc BUT with a fixed number of output size (that is saying, ...
1
vote
1answer
143 views

Training the generator in a GAN pair with back propagation

For the purposes of this question I am asking about training the generator, assume that training the discriminator is another topic. My understanding of generative adversarial networks is that you ...
2
votes
1answer
52 views

How is a neural network where the majority of inputs are 0 trained?

Consider AlexNet, which has 1000 output nodes, each of which classifies an image: The problem I have been having with training a neural network of similar proportions, is that it does what any ...
3
votes
4answers
93 views

Regression using neural network

I'd like to ask for any kind of assistance regarding the following problem: I was given the following training data: 100 numbers, each one is a parameter, they together define a number X(also given)....
3
votes
1answer
83 views

Why coupling coefficients in capsule neural networks can't be learned by back-propagation?

The paper Dynamic Routing Between Capsules uses the algorithm called "Dynamic Routing Between Capsules" to determine the coupling coefficients between capsules. Why it can't be done by ...
1
vote
0answers
100 views

When do you back-propagate errors through a neural network when using TD($\lambda$)?

I have a neural network that I'm want to use to self-play Connect Four. The neural network receives the board state and is to provide an estimate of the state's value. I would then, for each move, ...
2
votes
1answer
526 views

Is back propagation applied for each data point or for a batch of data points?

I am new to deep learning and trying to understand the concept of back propagation. I have a doubt on when the back propagation is applied. Assume that I have a training data set of 1000 images for ...
2
votes
2answers
99 views

Which neuron represents which part of the input?

In a neural network, each neuron represents some part of the input. For example, in the case of a MNIST digit, consider the stem of the number 9. Each neuron in the NN represents some part of this ...
0
votes
2answers
58 views

Backpropagation of neural nets with shared weight

I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. ...
1
vote
2answers
3k views

What is the derivative of the Leaky ReLU activation function?

I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the ...
18
votes
1answer
427 views

Are these two versions of back-propagation equivalent?

Just for fun, I am trying to develop a neural network. Now, for backpropagation I saw two techniques. The first one is used here and in many other places too. What it does is: It computes the ...
3
votes
0answers
41 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
3
votes
1answer
107 views

How do I calculate the partial derivative with respect to $x$?

I am trying to implement CNN using python Numpy. I searched so much, but all I found was for one filter with one channel for Convolution. Suppose we have an X as Image with this shape: ...
3
votes
1answer
1k views

How do I calculate the gradient of the hinge loss function?

With reference to the research paper entitled Sentiment Embeddings with Applications to Sentiment Analysis, I am trying to implement its sentiment ranking model in Python, for which I am required to ...
6
votes
1answer
6k views

How is the gradient calculated for the middle layer's weights?

I am trying to understand backpropagation. I used a simple neural network with one input $x$, one hidden layer $h$ and one output layer $y$, with weight $w_1$ connecting $x$ to $h$, and $w_2$ ...
2
votes
1answer
39 views

Should I compute the gradients with respect to the flatten layer in a convolutional neural network?

I'm trying to create a convolutional neural network without frameworks (such as PyTorch, TensorFlow, Keras, and so on) with Python. Here's a description of CNN taken from the Wikipedia article In ...
0
votes
0answers
53 views

Backward pass of CNN like Resnet: how to manually compute flops during backprop?

I've been trying to figure out how to compute the number of Flops in backward pass of ResNet. For forward pass, it seems straightforward: apply the conv filters to the input for each layer. But how ...
0
votes
0answers
18 views

Why do code implementations average the loss over a batch instead of finding the expected sample of that batch (using sampling probabilities)

Usually, our training objective over a batch is written in terms of the expected value of a sample in that batch such as $objective = E_{x \sim data} * log(P(x))$ But in the code implementations, ...
1
vote
1answer
32 views

How are the weights retained for filters for a particular class in a CNN?

I am new to CNN. What I have learned so far about the filters is that when we are giving a training example to our model, our model updates the weights by gradient descent to minimize the loss ...
1
vote
0answers
22 views

Class of functional equations that backpropagation can solve

There is a theorem that states that basically a neural network can approximate any function whatsoever. However, this does not mean that it can solve any equation. I have some notes where it states ...
2
votes
1answer
47 views

Does net with ReLU not learn when output < 0?

The derivative of ReLU is 0 if its output is lower than 0 - $d ReLU(x)/dReLU$ is $0$ if $x < 0$. Let's denote some net's output by $Out$, so if this net's last layer is ReLU then we get that $dOut/...
0
votes
1answer
42 views

Why are all weights of a neural net updated and not just the weights of the first layer

Why are all weights of a neural net updated and not only the weights of the first hidden layer? The error-influence of the prediction by the weights of a neural net is calculated using the chain rule....
2
votes
1answer
94 views

Backpropagation in Decoupled Neural Interfaces

I am attempting to create a fully decoupled feed-forward neural network by using decoupled neural interfaces as explained in the paper (https://arxiv.org/abs/1608.05343). As in the paper, the DNI is ...
1
vote
1answer
89 views

Simple three layer neural network with backpropagation is not approximating tanh function

I have this simple neural network in Python which I'm trying to use to aproximation tanh function. As inputs I have x - inputs to the function, and as outputs I want tanh(x) = y. I'm using sigmoid ...
8
votes
5answers
495 views

What is “backprop”?

What does "backprop" mean? Is the "backprop" term basically the same as "backpropagation" or does it have a different meaning?
2
votes
1answer
70 views

Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?

I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation. As per my knowledge, ...
2
votes
1answer
48 views

Why do we update all layers simultaneously while training a neural network?

Very deep models involve the composition of several functions or layers. The gradient tells how to update each parameter, under the assumption that the other layers do not change. In practice, we ...
22
votes
5answers
17k views

What is the time complexity for training a neural network using back-propagation?

Suppose that a NN contains $n$ hidden layers, $m$ training examples, $x$ features, and $n_i$ nodes in each layer. What is the time complexity to train this NN using back-propagation? I have a basic ...
4
votes
1answer
121 views

How to perform back propagation with different sized layers?

I'm developing my first neural network, using the well known MNIST database of handwritten digit. I want the NN to be able to classify a number from 0 to 9 given an image. My neural network consists ...
1
vote
1answer
56 views

Stochastic gradient descent does not behave as expected, even with different activation functions

I have been working on my own AI for a while now, trying to implemented SGD with momentum from scratch in python. After looking around and studying all the maths behind it, i finally managed to ...
2
votes
0answers
54 views

What is symbol-to-number differentiation?

I recently came across symbol-to-symbol and symbol-to-number differentiation, out of which symbol to symbol seemed fairly straightforward - the computational graph is extended to include gradient ...
2
votes
2answers
101 views

Why is it called back-propagation?

While looking at the mathematics of the back-propagation algorithm for a multi-layer perceptron, I noticed that in order to find the partial derivative of the cost function with respect to a weight (...
3
votes
1answer
54 views

How can I use a Hidden Markov Model to recognize images?

How could I use a 16x16 image as an input in a HMM? And at the same time how would I train it? Can I use backpropagation?
2
votes
2answers
50 views

How can I train a neural network if I don't have enough data?

I have created a neural network that is able to recognize images with the numbers 1-5. The issue is that I have a database of 16x5 images which ,unfortunately, is not proving enough as the neural ...
0
votes
1answer
67 views

Is my backpropagation code correct? [closed]

I am trying to implement the back-propagation algorithm for the following neural network. ...
2
votes
1answer
78 views

Different methods of calculating gradients of cost function(loss function)

We require to find the gradient of loss function(cost function) w.r.t to the weights to use optimization methods such as SGD or gradient descent. So far, I have come across two ways to compute the ...