# Questions tagged [backpropagation]

For questions related to the technique of backpropagation, whereby the loss, error, or correction signal calculated at the output of an artificial network output is fed back to the parameters in each layer of the network until the network's behavior converges to a training state within the required accuracy and reliability.

162 questions
Filter by
Sorted by
Tagged with
50 views

### What weights should I use while back-propagating?

I've started to learn about neural networks recently and I can't find the answer to this question. Let's assume there's a neural network (fig. 1) So if the loss function is: and the derivative is: ...
82 views

### How does adding a small change to an neuron's weighted input affect the overall cost?

I was reading the following book: http://neuralnetworksanddeeplearning.com/chap2.html and towards the end of equation 29, there is a paragraph that explains this: However I am unsure how the ...
1k views

### How do I calculate the gradient of the hinge loss function?

With reference to the research paper entitled Sentiment Embeddings with Applications to Sentiment Analysis, I am trying to implement its sentiment ranking model in Python, for which I am required to ...
83 views

### Why coupling coefficients in capsule neural networks can't be learned by back-propagation?

The paper Dynamic Routing Between Capsules uses the algorithm called "Dynamic Routing Between Capsules" to determine the coupling coefficients between capsules. Why it can't be done by ...
41 views

### How does backpropagation work in LSTMs?

After reading a lot of articles (for instance, this one Understanding LSTM Networks), I know that the long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used in ...
36 views

### How can I formulate a nonogram problem as a constraint satisfaction problem?

I've just started learning CSP and I find it quite exciting. Now I'm facing nonogram solving problem and I want to solve it using backtracking with CSP. The first problem that I face is that I cannot ...
93 views

### Regression using neural network

I'd like to ask for any kind of assistance regarding the following problem: I was given the following training data: 100 numbers, each one is a parameter, they together define a number X(also given)....
969 views

### How to perform neural network with output constraint?

Imagine a "simple" feedforward, fully connected neural network, with some input size, some number of hidden layers, and some # of neurons....etc BUT with a fixed number of output size (that is saying, ...
624 views

### How to calculate gradient of filter in convolution network

I have similar architecture like in image:CNN. I don't understand how to calculate gradient of filter F. I found these equations(source): Gradient and delta, where first equation calculate gradient ...
101 views

### Why is it called back-propagation?

While looking at the mathematics of the back-propagation algorithm for a multi-layer perceptron, I noticed that in order to find the partial derivative of the cost function with respect to a weight (...
50 views

### How can I train a neural network if I don't have enough data?

I have created a neural network that is able to recognize images with the numbers 1-5. The issue is that I have a database of 16x5 images which ,unfortunately, is not proving enough as the neural ...
45 views

### How can I train a neural network for another input set, without losing the learning of the previous input set?

I read this tutorial about backpropagation. So using this backpropagation we are training the neural network repeatedly for one input set, say [2,4], until we reach 100% accuracy of getting 1 as ...
526 views

### Is back propagation applied for each data point or for a batch of data points?

I am new to deep learning and trying to understand the concept of back propagation. I have a doubt on when the back propagation is applied. Assume that I have a training data set of 1000 images for ...
2k views

### What is the best XOR neural network configuration out there in terms of low error?

I'm trying to understand what would be the best neural network for implementing a XOR gate. I'm considering a neural network to be good if it can produce all the expected outcomes with the lowest ...
48 views

### Why do we update all layers simultaneously while training a neural network?

Very deep models involve the composition of several functions or layers. The gradient tells how to update each parameter, under the assumption that the other layers do not change. In practice, we ...
42 views

78 views

### Different methods of calculating gradients of cost function(loss function)

We require to find the gradient of loss function(cost function) w.r.t to the weights to use optimization methods such as SGD or gradient descent. So far, I have come across two ways to compute the ...
81 views

### What is the neuron-level math behind backpropagation for a neural network?

I am quite new in the AI field. I am trying to create a neural network, in a language (Dart) where I couldn't find examples or premade libraries or tutorials. I've tried looking online for a strictly "...
114 views

### How is gradient being calculated in Andrej Karpathy's pong code?

I was going through the code by Andrej Karpathy on reinforcement learning using a policy gradient. I have some questions from the code. Where is the logarithm of the probability being calculated? ...
51 views

### When and how to use a mix of loss functions for back-propagation?

I am trying to understand the best loss function to be used with a convolutional neural network. I came to know that we can mix two loss functions. Can any body share in what case was it done and how?
52 views

### How is a neural network where the majority of inputs are 0 trained?

Consider AlexNet, which has 1000 output nodes, each of which classifies an image: The problem I have been having with training a neural network of similar proportions, is that it does what any ...
146 views

### What is the difference between backpropagation and predictive coding?

Reading the high-level descriptions of backpropagation and predictive coding, they don't sound so drastically different. What is the key difference between these techniques? I am currently reading ...
50 views

### What is the use of the $\epsilon$ term in this back-propagation equation?

I am currently looking at different documents to understand back-propagation, mainly at this document. Now, at page 3, there is the $\epsilon$ symbol involved: While I understand the main part of the ...
961 views

### Should the weights of a neural network be updated after each example or at the end of the batch? [duplicate]

Should the weights of a neural network be updated after each example or at the end of the batch? Do I need a normalization factor in the second case?
569 views

### How to change the backward pass for an LSTM layer that outputs to another LSTM layer?

I am currently trying to understand the mathematics in Ger's paper Long Short-Term Memory in Recurrent Neural Networks. I have found the document clear and readable so far. On pg. 21 of the pdf (pg. ...
35 views

### Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?

I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, ...
54 views

### What is symbol-to-number differentiation?

I recently came across symbol-to-symbol and symbol-to-number differentiation, out of which symbol to symbol seemed fairly straightforward - the computational graph is extended to include gradient ...
70 views

### Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?

I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation. As per my knowledge, ...
63 views

### Why gradients are so small in deep learning?

The learning rate in my model is 0.00001 and the gradients of the model is within the distribution of [-0.0001, 0.0001]. Is it ...
30 views

### How is the gradient with respect to weights derived in batch normalization?

This is a cross-post, as I didn't get any answers on Stats SE and I am hoping that it gets more attention here. At the bottom of page 2 of the paper L2 Regularization versus Batch and Weight ...
136 views

### How can I implement derivative of softmax function for matrices in Python? [closed]

I have trouble understanding how to implement derivative of softmax function. Here is what I tried: ...
14 views

### yolo output and how to define labels for backpropogation on it

I want to build the yolo architecture in keras but can't understand the basic idea behind the training of the yolo, like how to define the labels for whether there is no object there what we have to ...
21 views

### Calculation of Neural network biases in backpropagation

While learning neural networks I've found a basic Python working example to play with. It has 3 input nodes, 4 nodes in a hidden layer, 1 output node. 5 data sets for training. The initial code is ...
42 views

### How does a Bidirectional RNN work?

Could it be possible to reach a similar output via feeding a unidirectional network with the original data and the data played backwards?
99 views

### Which neuron represents which part of the input?

In a neural network, each neuron represents some part of the input. For example, in the case of a MNIST digit, consider the stem of the number 9. Each neuron in the NN represents some part of this ...
30 views

### Use of backpropagation for weight updates in a combination of 2 neural networks

Every neural network updates its weights through back-propagation. How is back-propagation used for updating weights in a combination of 2 or more neural networks (e.g.:CNN-LSTM, GAN-CNN, etc.). For ...
103 views

### Update of weights in Recurrent Neural Network through back propagation

How does Recurrent Neural Network updates its weights and bias through backpropagation? Is time taken into account while updating the weights of a RNN using Backpropagation through time(BPTT)?"
51 views

### Using features extracted from a CNN as convolutional filter

I'm a bit confused about this. Assume I have a CNN network with two branches: Top Bottom The top branch outputs a feature vector of shape 1x1x1x10 (batch, h, w, c) The bottom branch outputs a ...
118 views

### How to determine the size of biases? [closed]

I'm new to the world of machine learning. My question is how can I determine the size of the biases in a neural network (with backpropagation algorithm)? Currently, I have a 2-layer neural network (1 ...
120 views

### Data prepared to linear regression. Can I use it with backpropagation?

I'm studying a Master's Degree in Artificial Intelligence and I need to learn how to use the Java Neural Network Simulator, JavaNNS, program. In one practice I have to build a neural network to use ...
47 views

### Recommendations on which architecture to use to guess appointment

I'm currently developping an application which allows psychologists to manage their schedule and budget. As a proof of concept, I would like to create an intelligent appointment service. There can be ...
94 views

### Backpropagation in Decoupled Neural Interfaces

I am attempting to create a fully decoupled feed-forward neural network by using decoupled neural interfaces as explained in the paper (https://arxiv.org/abs/1608.05343). As in the paper, the DNI is ...
460 views

### What is the relation between back-propagation and reinforcement learning?

What is the relation between back-propagation and reinforcement learning?
I am reading about backpropagation and I wonder why I have to backpropagate. For example, I would update the network by randomly choosing a weight to change, $w$. I would have $X$ and $y$. Then, I ...