Questions tagged [backpropagation]

For questions related to the technique of backpropagation, whereby the loss, error, or correction signal calculated at the output of an artificial network output is fed back to the parameters in each layer of the network until the network's behavior converges to a training state within the required accuracy and reliability.

Filter by
Sorted by
Tagged with
2
votes
0answers
32 views

Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
0
votes
0answers
23 views

How to find the derivative of a dynamic neuron model, which depends on previous states of the neuron?

This is the equation where n denotes the current state, (n-1) denotes the state in the previous step etc. And to do back-propagation I need to find partial derivatives over each of the variables. For ...
0
votes
0answers
20 views

Finding the energy function given update rule of a single layer non-linear neural network

Consider the network with N neurons, each of which takes a $2 \times k$ input specified by the tuple $(\vec c_t, \vec \theta_t)$ to produce output $\vec{R}_t$ through an update rule on the pairwise ...
1
vote
3answers
98 views

Why does a neuron in a multi-layer network need several input connections?

For example, if I have the following architecture: Each neuron in the hidden layer has a connection from each one in the input layer. 3 x 1 Input Matrix and a 4 x 3 weight matrix (for the ...
0
votes
0answers
55 views

Backpropagation implementation with Java

I've been trying to implement a Multilayer Perceptron Network using java language with the ultimate goal of creating and teaching a neural network to recognize handwritten digits. Pretty simple and ...
4
votes
0answers
68 views

What should I do with the flatten layer during back-propagation?

I'm creating a CNN network without other frameworks such as PyTorch, Keras, Tensorflow, and so on. During the forward pass, the Flatten layer reshapes the previous ...
0
votes
1answer
59 views

Is my understanding of back-propogation correct?

I am trying to learn backpropagation and this is what I know so far. To update the weights of the neural network you have to figure out the partial derivative of each of the parameters on the loss ...
0
votes
0answers
40 views

Why is it that having a duplicate in features set makes training to work bad

I'm defining a deep network to emulate a multitarget regression. When I costruct my training set, I take information from a graph; without going into too much detail, it could happen that I take 2 ...
1
vote
0answers
56 views

I need help understanding general back propagation algorithm

In section 6.5.6 of the book Deep Learning by Ian et. al. general backpropagation algorithm is described as: The back-propagation algorithm is very simple. To compute the gradient of some scalar z ...
3
votes
0answers
41 views

How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
3
votes
1answer
107 views

How do I calculate the partial derivative with respect to $x$?

I am trying to implement CNN using python Numpy. I searched so much, but all I found was for one filter with one channel for Convolution. Suppose we have an X as Image with this shape: ...
0
votes
0answers
53 views

Backward pass of CNN like Resnet: how to manually compute flops during backprop?

I've been trying to figure out how to compute the number of Flops in backward pass of ResNet. For forward pass, it seems straightforward: apply the conv filters to the input for each layer. But how ...
0
votes
2answers
58 views

Backpropagation of neural nets with shared weight

I am trying to understand the mathematics behind the forward and backward propagation of neural nets. To make myself more comfortable, I am testing myself with an arbitrarily chosen neural network. ...
0
votes
0answers
18 views

Why do code implementations average the loss over a batch instead of finding the expected sample of that batch (using sampling probabilities)

Usually, our training objective over a batch is written in terms of the expected value of a sample in that batch such as $objective = E_{x \sim data} * log(P(x))$ But in the code implementations, ...
2
votes
1answer
39 views

Should I compute the gradients with respect to the flatten layer in a convolutional neural network?

I'm trying to create a convolutional neural network without frameworks (such as PyTorch, TensorFlow, Keras, and so on) with Python. Here's a description of CNN taken from the Wikipedia article In ...
1
vote
0answers
22 views

Class of functional equations that backpropagation can solve

There is a theorem that states that basically a neural network can approximate any function whatsoever. However, this does not mean that it can solve any equation. I have some notes where it states ...
2
votes
1answer
47 views

Does net with ReLU not learn when output < 0?

The derivative of ReLU is 0 if its output is lower than 0 - $d ReLU(x)/dReLU$ is $0$ if $x < 0$. Let's denote some net's output by $Out$, so if this net's last layer is ReLU then we get that $dOut/...
0
votes
1answer
42 views

Why are all weights of a neural net updated and not just the weights of the first layer

Why are all weights of a neural net updated and not only the weights of the first hidden layer? The error-influence of the prediction by the weights of a neural net is calculated using the chain rule....
2
votes
1answer
48 views

Why do we update all layers simultaneously while training a neural network?

Very deep models involve the composition of several functions or layers. The gradient tells how to update each parameter, under the assumption that the other layers do not change. In practice, we ...
2
votes
0answers
54 views

What is symbol-to-number differentiation?

I recently came across symbol-to-symbol and symbol-to-number differentiation, out of which symbol to symbol seemed fairly straightforward - the computational graph is extended to include gradient ...
4
votes
1answer
121 views

How to perform back propagation with different sized layers?

I'm developing my first neural network, using the well known MNIST database of handwritten digit. I want the NN to be able to classify a number from 0 to 9 given an image. My neural network consists ...
2
votes
2answers
101 views

Why is it called back-propagation?

While looking at the mathematics of the back-propagation algorithm for a multi-layer perceptron, I noticed that in order to find the partial derivative of the cost function with respect to a weight (...
1
vote
1answer
56 views

Stochastic gradient descent does not behave as expected, even with different activation functions

I have been working on my own AI for a while now, trying to implemented SGD with momentum from scratch in python. After looking around and studying all the maths behind it, i finally managed to ...
3
votes
1answer
54 views

How can I use a Hidden Markov Model to recognize images?

How could I use a 16x16 image as an input in a HMM? And at the same time how would I train it? Can I use backpropagation?
2
votes
2answers
50 views

How can I train a neural network if I don't have enough data?

I have created a neural network that is able to recognize images with the numbers 1-5. The issue is that I have a database of 16x5 images which ,unfortunately, is not proving enough as the neural ...
0
votes
1answer
67 views

Is my backpropagation code correct? [closed]

I am trying to implement the back-propagation algorithm for the following neural network. ...
2
votes
1answer
78 views

Different methods of calculating gradients of cost function(loss function)

We require to find the gradient of loss function(cost function) w.r.t to the weights to use optimization methods such as SGD or gradient descent. So far, I have come across two ways to compute the ...
1
vote
1answer
32 views

How are the weights retained for filters for a particular class in a CNN?

I am new to CNN. What I have learned so far about the filters is that when we are giving a training example to our model, our model updates the weights by gradient descent to minimize the loss ...
1
vote
0answers
32 views

Why do DeconvNet use ReLU in the backward pass?

Why does DeconvNet (Zeiler, 2014) use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply ...
4
votes
0answers
21 views

How do weights changes handles during back-propagation when there are unknown labels

I have a question about how weights are updated during back-propagation for some of my samples that have unknown labels (please note, unknown, not missing). The reason they are unknown is because this ...
3
votes
0answers
36 views

How can I formulate a nonogram problem as a constraint satisfaction problem?

I've just started learning CSP and I find it quite exciting. Now I'm facing nonogram solving problem and I want to solve it using backtracking with CSP. The first problem that I face is that I cannot ...
1
vote
1answer
89 views

Simple three layer neural network with backpropagation is not approximating tanh function

I have this simple neural network in Python which I'm trying to use to aproximation tanh function. As inputs I have x - inputs to the function, and as outputs I want tanh(x) = y. I'm using sigmoid ...
0
votes
0answers
53 views

Derivation of regularized cost function w.r.t activation and bias

In regularzied cost function a L2 regularization cost has been added. Here we have already calculated cross entropy cost w.r.t $A, W$. As mentioned in the regularization notebook (see below) in ...
2
votes
1answer
70 views

Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?

I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation. As per my knowledge, ...
1
vote
0answers
40 views

Which activation functions can lead to the vanishing gradient problem?

From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem. What other activation functions can lead to the ...
2
votes
2answers
42 views

How to determine the target value when using ReLU as activation function?

Consider the following simple neural network with only one neuron. The input is $x_1$ and $y_2$, where $-250 < x < 250$ and $-250 < y < 250$ The weights of the only neuron are $w_1$ and $...
4
votes
1answer
862 views

What is the purpose of the batch size in neural networks?

Why is a batch size needed to update the weights of a neural network? According to that Youtube Video from 3B1B, the weights are updated by calculating the error between expectation and outcome of ...
2
votes
0answers
63 views

Why gradients are so small in deep learning?

The learning rate in my model is 0.00001 and the gradients of the model is within the distribution of [-0.0001, 0.0001]. Is it ...
2
votes
0answers
30 views

How is the gradient with respect to weights derived in batch normalization?

This is a cross-post, as I didn't get any answers on Stats SE and I am hoping that it gets more attention here. At the bottom of page 2 of the paper L2 Regularization versus Batch and Weight ...
1
vote
0answers
28 views

Function to update weights in back-propagation

I am trying to wrap my head around how weights get updated during back propagation. I've been going through a school book and I have the following setup for an ANN with 1 hidden layer, a couple of ...
0
votes
0answers
13 views

How to perform PGD on a pretrained CNN?

I have a pretrained CNN model using the keras library. I now need to perform a Projected Gradient Descent (PGD) to develop some adversarial examples. To do this, I will need to perform a gradient ...
0
votes
1answer
22 views

Positive bias causes the calculation of incorrect gradients

I have a data set with a positive bias (an image, where the values range from 0 to 1), that seems to be causing my network to calculate incorrect gradients. If I just use the raw image as input, of ...
0
votes
0answers
28 views

Backpropagation and PID

In backpropagation, the update rule for the weights is based on the derivative of some loss function. This is similar to the "proportional" aspect of a PID loop controller, where some control variable ...
2
votes
0answers
136 views

How can I implement derivative of softmax function for matrices in Python? [closed]

I have trouble understanding how to implement derivative of softmax function. Here is what I tried: ...
1
vote
1answer
56 views

How is back-propagation useful in neural networks?

I am reading about backpropagation and I wonder why I have to backpropagate. For example, I would update the network by randomly choosing a weight to change, $w$. I would have $X$ and $y$. Then, I ...
1
vote
1answer
30 views

What is the difference between batches in deep Q learning and supervised learning?

How is the batch loss calculated in both DQNs and simple classifiers? From what I understood, in a classifier, a common method is that you sample a mini-batch, calculate the loss for every example, ...
3
votes
3answers
287 views

What's the function that SGD takes to calculate the gradient?

I'm struggling to fully understand the stochastic gradient descent algorithm. I know that gradient descent allows you to find the local minimum of a function. What I don't know is what exactly that ...
4
votes
1answer
71 views

How can a DQN backpropagate its loss?

I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I ...
2
votes
1answer
81 views

What is the neuron-level math behind backpropagation for a neural network?

I am quite new in the AI field. I am trying to create a neural network, in a language (Dart) where I couldn't find examples or premade libraries or tutorials. I've tried looking online for a strictly "...
1
vote
2answers
122 views

How can we compute the gradient of max pooling with overlapping regions?

Studying CNN Back-propagation I can't understand how can we compute the gradient of max pooling with overlapping regions ? That's also a question from this quiz and can be also found on this book .