Questions tagged [backpropagation]

For questions related to the technique of backpropagation, whereby the loss, error, or correction signal calculated at the output of an artificial network output is fed back to the parameters in each layer of the network until the network's behavior converges to a training state within the required accuracy and reliability.

Filter by
Sorted by
Tagged with
2
votes
0answers
21 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
3
votes
1answer
87 views
+50

How do I calculate the partial derivative with respect to $x$?

I am trying to implement CNN using python Numpy. I searched so much, but all I found was for one filter with one channel for Convolution. Suppose we have an X as Image with this shape: ...
0
votes
0answers
18 views

Backward pass of CNN like Resnet: how to manually compute flops during backprop?

I've been trying to figure out how to compute the number of Flops in backward pass of ResNet. For forward pass, it seems straightforward: apply the conv filters to the input for each layer. But how ...
0
votes
2answers
38 views

Backpropagation of neural nets with shared weight

I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. ...
0
votes
0answers
10 views

Why do code implementations average the loss over a batch instead of finding the expected sample of that batch (using sampling probabilities)

Usually, our training objective over a batch is written in terms of the expected value of a sample in that batch such as $objective = E_{x \sim data} * log(P(x))$ But in the code implementations, ...
2
votes
1answer
28 views

Should I compute the gradients with respect to the flatten layer in a convolutional neural network?

I'm trying to create a convolutional neural network without frameworks (such as PyTorch, TensorFlow, Keras, and so on) with Python. Here's a description of CNN taken from the Wikipedia article In ...
1
vote
0answers
9 views

Class of functional equations that backpropagation can solve

There is a theorem that states that basically a neural network can approximate any function whatsoever. However, this does not mean that it can solve any equation. I have some notes where it states ...
2
votes
1answer
36 views

Does net with ReLU not learn when output < 0?

The derivative of ReLU is 0 if its output is lower than 0 - $d ReLU(x)/dReLU$ is $0$ if $x < 0$. Let's denote some net's output by $Out$, so if this net's last layer is ReLU then we get that $dOut/...
0
votes
1answer
30 views

Why are all weights of a neural net updated and not just the weights of the first layer

Why are all weights of a neural net updated and not only the weights of the first hidden layer? The error-influence of the prediction by the weights of a neural net is calculated using the chain rule....
2
votes
1answer
34 views

Why do we update all layers simultaneously while training a neural network?

Very deep models involve the composition of several functions or layers. The gradient tells how to update each parameter, under the assumption that the other layers do not change. In practice, we ...
2
votes
0answers
34 views

What is symbol-to-number differentiation?

I recently came across symbol-to-symbol and symbol-to-number differentiation, out of which symbol to symbol seemed fairly straightforward - the computational graph is extended to include gradient ...
4
votes
1answer
114 views

How to perform back propagation with different sized layers?

I'm developing my first neural network, using the well known MNIST database of handwritten digit. I want the NN to be able to classify a number from 0 to 9 given an image. My neural network consists ...
2
votes
2answers
88 views

Why is it called back-propagation?

While looking at the mathematics of the back-propagation algorithm for a multi-layer perceptron, I noticed that in order to find the partial derivative of the cost function with respect to a weight (...
1
vote
1answer
47 views

Stochastic gradient descent does not behave as expected, even with different activation functions

I have been working on my own AI for a while now, trying to implemented SGD with momentum from scratch in python. After looking around and studying all the maths behind it, i finally managed to ...
2
votes
1answer
43 views

How can I use a Hidden Markov Model to recognize images?

How could I use a 16x16 image as an input in a HMM? And at the same time how would I train it? Can I use backpropagation?
1
vote
2answers
39 views

How can I train a neural network if I don't have enough data?

I have created a neural network that is able to recognize images with the numbers 1-5. The issue is that I have a database of 16x5 images which ,unfortunately, is not proving enough as the neural ...
0
votes
1answer
62 views

Is my backpropagation code correct? [closed]

I am trying to implement the back-propagation algorithm for the following neural network. ...
2
votes
1answer
60 views

Different methods of calculating gradients of cost function(loss function)

We require to find the gradient of loss function(cost function) w.r.t to the weights to use optimization methods such as SGD or gradient descent. So far, I have come across two ways to compute the ...
1
vote
1answer
25 views

How are the weights retained for filters for a particular class in a CNN?

I am new to CNN. What I have learned so far about the filters is that when we are giving a training example to our model, our model updates the weights by gradient descent to minimize the loss ...
1
vote
0answers
27 views

Why do DeconvNet use ReLU in the backward pass?

Why does DeconvNet (Zeiler, 2014) use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply ...
4
votes
0answers
14 views

How do weights changes handles during back-propagation when there are unknown labels

I have a question about how weights are updated during back-propagation for some of my samples that have unknown labels (please note, unknown, not missing). The reason they are unknown is because this ...
3
votes
0answers
26 views

How can I formulate a nonogram problem as a constraint satisfaction problem?

I've just started learning CSP and I find it quite exciting. Now I'm facing nonogram solving problem and I want to solve it using backtracking with CSP. The first problem that I face is that I cannot ...
1
vote
1answer
61 views

Simple three layer neural network with backpropagation is not approximating tanh function

I have this simple neural network in Python which I'm trying to use to aproximation tanh function. As inputs I have x - inputs to the function, and as outputs I want tanh(x) = y. I'm using sigmoid ...
0
votes
0answers
36 views

Derivation of regularized cost function w.r.t activation and bias

In regularzied cost function a L2 regularization cost has been added. Here we have already calculated cross entropy cost w.r.t $A, W$. As mentioned in the regularization notebook (see below) in ...
2
votes
1answer
60 views

Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?

I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation. As per my knowledge, ...
1
vote
0answers
25 views

Which activation functions can lead to the vanishing gradient problem?

From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem. What other activation functions can lead to the ...
2
votes
2answers
35 views

How to determine the target value when using ReLU as activation function?

Consider the following simple neural network with only one neuron. The input is $x_1$ and $y_2$, where $-250 < x < 250$ and $-250 < y < 250$ The weights of the only neuron are $w_1$ and $...
4
votes
1answer
841 views

What is the purpose of the batch size in neural networks?

Why is a batch size needed to update the weights of a neural network? According to that Youtube Video from 3B1B, the weights are updated by calculating the error between expectation and outcome of ...
2
votes
0answers
58 views

Why gradients are so small in deep learning?

The learning rate in my model is 0.00001 and the gradients of the model is within the distribution of [-0.0001, 0.0001]. Is it ...
2
votes
0answers
23 views

How is the gradient with respect to weights derived in batch normalization?

This is a cross-post, as I didn't get any answers on Stats SE and I am hoping that it gets more attention here. At the bottom of page 2 of the paper L2 Regularization versus Batch and Weight ...
1
vote
0answers
23 views

Function to update weights in back-propagation

I am trying to wrap my head around how weights get updated during back propagation. I've been going through a school book and I have the following setup for an ANN with 1 hidden layer, a couple of ...
0
votes
0answers
8 views

How to perform PGD on a pretrained CNN?

I have a pretrained CNN model using the keras library. I now need to perform a Projected Gradient Descent (PGD) to develop some adversarial examples. To do this, I will need to perform a gradient ...
0
votes
1answer
18 views

Positive bias causes the calculation of incorrect gradients

I have a data set with a positive bias (an image, where the values range from 0 to 1), that seems to be causing my network to calculate incorrect gradients. If I just use the raw image as input, of ...
0
votes
0answers
23 views

Backpropagation and PID

In backpropagation, the update rule for the weights is based on the derivative of some loss function. This is similar to the "proportional" aspect of a PID loop controller, where some control variable ...
2
votes
0answers
99 views

How can I implement derivative of softmax function for matrices in Python?

I have trouble understanding how to implement derivative of softmax function. Here is what I tried: ...
1
vote
1answer
52 views

How is back-propagation useful in neural networks?

I am reading about backpropagation and I wonder why I have to backpropagate. For example, I would update the network by randomly choosing a weight to change, $w$. I would have $X$ and $y$. Then, I ...
0
votes
1answer
21 views

What is the difference between batches in deep Q learning and supervised learning?

How is the batch loss calculated in both DQNs and simple classifiers? From what I understood, in a classifier, a common method is that you sample a mini-batch, calculate the loss for every example, ...
3
votes
3answers
234 views

What's the function that SGD takes to calculate the gradient?

I'm struggling to fully understand the stochastic gradient descent algorithm. I know that gradient descent allows you to find the local minimum of a function. What I don't know is what exactly that ...
4
votes
1answer
47 views

How can a DQN backpropagate its loss?

I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I ...
2
votes
1answer
76 views

What is the neuron-level math behind backpropagation for a neural network?

I am quite new in the AI field. I am trying to create a neural network, in a language (Dart) where I couldn't find examples or premade libraries or tutorials. I've tried looking online for a strictly "...
1
vote
2answers
97 views

How can we compute the gradient of max pooling with overlapping regions?

Studying CNN Back-propagation I can't understand how can we compute the gradient of max pooling with overlapping regions ? That's also a question from this quiz and can be also found on this book .
3
votes
4answers
76 views

Regression using neural network

I'd like to ask for any kind of assistance regarding the following problem: I was given the following training data: 100 numbers, each one is a parameter, they together define a number X(also given)....
2
votes
1answer
36 views

What would be the implications of mistakenly adding bias after the activation function?

I was looking at the source code for a personal project neural network implementation, and the bias for each node was mistakenly applied after the activation function. The output of each node was ...
2
votes
1answer
40 views

How can I train a neural network for another input set, without losing the learning of the previous input set?

I read this tutorial about backpropagation. So using this backpropagation we are training the neural network repeatedly for one input set, say [2,4], until we reach 100% accuracy of getting 1 as ...
1
vote
0answers
24 views

Is this TensorFlow implementation of partial derivative of the cost with respect to the bias correct?

I have a neural network for MNIST classification which I am hard coding using TensorFlow 2.0. The neural network has an input layer consisting of 784 neurons (28 * 28), one hidden layer having "...
3
votes
1answer
98 views

How does the neural-network know how to tweak weights for a specific neuron?

I know backpropagation uses cost and gradient descent to tweak the weights to minimize the cost. But how does it know which weights to give more weight to in the first place? Is there something inside ...
2
votes
0answers
10 views

yolo output and how to define labels for backpropogation on it

I want to build the yolo architecture in keras but can't understand the basic idea behind the training of the yolo, like how to define the labels for whether there is no object there what we have to ...
2
votes
0answers
18 views

Calculation of Neural network biases in backpropagation

While learning neural networks I've found a basic Python working example to play with. It has 3 input nodes, 4 nodes in a hidden layer, 1 output node. 5 data sets for training. The initial code is ...
3
votes
1answer
59 views

What kind of data structures are needed to efficiently do back-propagation in a feedforward neural network?

In a feed-forward neural network, in order to efficiently do backpropagation, what kind of data structure is needed? I know the weights can just be stored in an array, and you need pointers of some ...
2
votes
1answer
112 views

How is gradient being calculated in Andrej Karpathy's pong code?

I was going through the code by Andrej Karpathy on reinforcement learning using a policy gradient. I have some questions from the code. Where is the logarithm of the probability being calculated? ...