Questions tagged [backpropagation]

For questions related to the technique of backpropagation, whereby the loss, error, or correction signal calculated at the output of an artificial network output is fed back to the parameters in each layer of the network until the network's behavior converges to a training state within the required accuracy and reliability.

47 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
2answers
73 views

Why do very deep non resnet architectures perform worse compared to shallower ones for the same iteration? Shouldn't they just train slower?

My understanding of the vanishing gradient problem in deep networks is that as backprop progresses through the layers the gradients become small, and thus training progresses slower. I'm having a hard ...
5
votes
0answers
127 views

How to design 4D Deep Recurrent Neural Networks using Tensorflow?

I want to design a simple model that predicts the movement of coordinates with RNNs. In a typical three-dimensional LSTM model, one feature is encoded as one hot encoding, and the ...
4
votes
0answers
13 views

How do weights changes handles during back-propagation when there are unknown labels

I have a question about how weights are updated during back-propagation for some of my samples that have unknown labels (please note, unknown, not missing). The reason they are unknown is because this ...
4
votes
1answer
83 views

Backpropagation equation for a variant on the usual Linear Neuron architecture

Recently I encountered a variant on the normal linear neural layer architecture: Instead of $Z = XW + B$, we now have $Z = (X-A)W + B$. So we have a 'pre-bias' $A$ that affects the activation of the ...
3
votes
0answers
570 views

How to calculate gradient of filter in convolution network

I have similar architecture like in image:CNN. I don't understand how to calculate gradient of filter F. I found these equations(source): Gradient and delta, where first equation calculate gradient ...
2
votes
0answers
19 views

How can I formulate a nonogram problem as a constraint satisfaction problem?

I've just started learning CSP and I find it quite exciting. Now I'm facing nonogram solving problem and I want to solve it using backtracking with CSP. The first problem that I face is that I cannot ...
2
votes
1answer
57 views

Why is my derivation of the back-propagation equations inconsistent with Andrew Ng's slides from Coursera?

I am using the cross-entropy cost function to calculate its derivatives using different variables $Z, W$ and $b$ at different instances. Please refer image below for calculation. As per my knowledge, ...
2
votes
0answers
53 views

Why gradients are so small in deep learning?

The learning rate in my model is 0.00001 and the gradients of the model is within the distribution of [-0.0001, 0.0001]. Is it ...
2
votes
0answers
20 views

How is the gradient with respect to weights derived in batch normalization?

This is a cross-post, as I didn't get any answers on Stats SE and I am hoping that it gets more attention here. At the bottom of page 2 of the paper L2 Regularization versus Batch and Weight ...
2
votes
0answers
66 views

How can I implement derivative of softmax function for matrices in Python?

I have trouble understanding how to implement derivative of softmax function. Here is what I tried: ...
2
votes
1answer
43 views

Regression using neural network

I'd like to ask for any kind of assistance regarding the following problem: I was given the following training data: 100 numbers, each one is a parameter, they together define a number X(also given)....
2
votes
0answers
17 views

Calculation of Neural network biases in backpropagation

While learning neural networks I've found a basic Python working example to play with. It has 3 input nodes, 4 nodes in a hidden layer, 1 output node. 5 data sets for training. The initial code is ...
2
votes
0answers
40 views

How does a Bidirectional RNN work?

Could it be possible to reach a similar output via feeding a unidirectional network with the original data and the data played backwards?
2
votes
2answers
88 views

Which neuron represents which part of the input?

In a neural network, each neuron represents some part of the input. For example, in the case of a MNIST digit, consider the stem of the number 9. Each neuron in the NN represents some part of this ...
2
votes
0answers
97 views

Update of weights in Recurrent Neural Network through back propagation

How does Recurrent Neural Network updates its weights and bias through backpropagation? Is time taken into account while updating the weights of a RNN using Backpropagation through time(BPTT)?"
2
votes
2answers
726 views

How to perform neural network with output constraint?

Imagine a "simple" feedforward, fully connected neural network, with some input size, some number of hidden layers, and some # of neurons....etc BUT with a fixed number of output size (that is saying, ...
2
votes
0answers
112 views

Data prepared to linear regression. Can I use it with backpropagation?

I'm studying a Master's Degree in Artificial Intelligence and I need to learn how to use the Java Neural Network Simulator, JavaNNS, program. In one practice I have to build a neural network to use ...
2
votes
0answers
46 views

Recommendations on which architecture to use to guess appointment

I'm currently developping an application which allows psychologists to manage their schedule and budget. As a proof of concept, I would like to create an intelligent appointment service. There can be ...
2
votes
1answer
89 views

Backpropagation in Decoupled Neural Interfaces

I am attempting to create a fully decoupled feed-forward neural network by using decoupled neural interfaces as explained in the paper (https://arxiv.org/abs/1608.05343). As in the paper, the DNI is ...
1
vote
1answer
16 views

How are the weights retained for filters for a particular class in a CNN?

I am new to CNN. What I have learned so far about the filters is that when we are giving a training example to our model, our model updates the weights by gradient descent to minimize the loss ...
1
vote
0answers
23 views

Why do DeconvNet use ReLU in the backward pass?

Why does DeconvNet (Zeiler, 2014) use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply ...
1
vote
1answer
41 views

Simple three layer neural network with backpropagation is not approximating tanh function

I have this simple neural network in Python which I'm trying to use to aproximation tanh function. As inputs I have x - inputs to the function, and as outputs I want tanh(x) = y. I'm using sigmoid ...
1
vote
0answers
25 views

Which activation functions can lead to the vanishing gradient problem?

From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem. What other activation functions can lead to the ...
1
vote
0answers
22 views

Is this TensorFlow implementation of partial derivative of the cost with respect to the bias correct?

I have a neural network for MNIST classification which I am hard coding using TensorFlow 2.0. The neural network has an input layer consisting of 784 neurons (28 * 28), one hidden layer having "...
1
vote
0answers
9 views

yolo output and how to define labels for backpropogation on it

I want to build the yolo architecture in keras but can't understand the basic idea behind the training of the yolo, like how to define the labels for whether there is no object there what we have to ...
1
vote
0answers
33 views

Understanding the partial derivative with respect to the weight matrix and bias

Say we have the layer $X W + b = Y$. I want to get $\frac{dL}{dW}$ and we assume I have $\frac{dL}{dY}$. So all I need is to find $\frac{dY}{dW}$. I know that it should be $X^T\frac{dL}{dY}$ but don'...
1
vote
0answers
29 views

How does a single neuron in hidden layer affect training accuracy

I'm currently a student learning about AI Networks. I've came across a statement in one of my Professor's books that a FFBP (Feed-Forward Back-Propagation) Neural Network with a single hidden layer ...
1
vote
0answers
50 views

How to train and update weights of filters

I have some problems with training CNN :( For example: Input 6x6x3, 1 core 3x3x3, output = 4x4x1 => pool: 2x2x1 By backpropagation I calculated deltas for output. This tutor and other tutors are ...
1
vote
0answers
95 views

Difficulties to implement the layer-wise relevance propagation in MATLAB

I'm having serious issues with the implementation of the LRP algorithm for neural networks in MATLAB. The challenge is to implement the equations correctly. I'm trying to implement the deep-Taylor $\...
1
vote
0answers
53 views

How are the weights between the input and hidden layer updated in a 3 layer neural network?

Consider a feed-forward neural network with one hidden layer. How are the weights between the input and hidden layer updated, after the weights between the hidden layer and output layer are updated?
1
vote
0answers
28 views

Use of backpropagation for weight updates in a combination of 2 neural networks

Every neural network updates its weights through back-propagation. How is back-propagation used for updating weights in a combination of 2 or more neural networks (e.g.:CNN-LSTM, GAN-CNN, etc.). For ...
1
vote
0answers
49 views

Using features extracted from a CNN as convolutional filter

I'm a bit confused about this. Assume I have a CNN network with two branches: Top Bottom The top branch outputs a feature vector of shape 1x1x1x10 (batch, h, w, c) The bottom branch outputs a ...
1
vote
0answers
29 views

Has anyone investigated iteration awareness beyond RNN and LSTM?

This question considers the convergence of an artificial networks (MLPs, RNNs, LSTM nets, CNNs) over time or over the course of epochs made up of iterations through training examples. In this ...
1
vote
0answers
317 views

How does backpropagation work on a custom loss function whose components have magnitudes of different orders?

I want to use a custom loss function which is a weighted combination of l1 and DSSIM losses. The DSSIM loss is limited between 0 and 0.5 where as the l1 loss can be orders of magnitude greater and is ...
1
vote
0answers
89 views

When do you back-propagate errors through a Neural Network when using TD Lambda

I have a Neural Network that I'm want to use to self-play Connect Four. The neural network receives the board state and is to provide an estimate of the states desirability. I would then, for each ...
1
vote
1answer
501 views

How to find partial derivative of softmax w.r.t logits in python

i have trouble implementing back propogation for multi class classification of CIFAR10 dataset My neural network has 2 layers forward propagation X -> L1 -> L2 weights W are initialized as random ...
0
votes
0answers
34 views

Derivation of regularized cost function w.r.t activation and bias

In regularzied cost function a L2 regularization cost has been added. Here we have already calculated cross entropy cost w.r.t $A, W$. As mentioned in the regularization notebook (see below) in ...
0
votes
0answers
22 views

Function to update weights in back-propagation

I am trying to wrap my head around how weights get updated during back propagation. I've been going through a school book and I have the following setup for an ANN with 1 hidden layer, a couple of ...
0
votes
0answers
7 views

How to perform PGD on a pretrained CNN?

I have a pretrained CNN model using the keras library. I now need to perform a Projected Gradient Descent (PGD) to develop some adversarial examples. To do this, I will need to perform a gradient ...
0
votes
1answer
18 views

Positive bias causes the calculation of incorrect gradients

I have a data set with a positive bias (an image, where the values range from 0 to 1), that seems to be causing my network to calculate incorrect gradients. If I just use the raw image as input, of ...
0
votes
0answers
22 views

Backpropagation and PID

In backpropagation, the update rule for the weights is based on the derivative of some loss function. This is similar to the "proportional" aspect of a PID loop controller, where some control variable ...
0
votes
0answers
33 views

Not able to properly tune Neural Network via Back Propagation properly

I have a custom code Neural Network(not using keras or any package...Trying to learn the essence of Neural Network from scratch)... Code can be found here I have the per iteration training output(<...
0
votes
0answers
42 views

Confused about NeuralODE

I am a bit confused about NeuralODE and I want to make sure that what I understood so far is correct. Assume we have (for simplicity) 2 data points $z_0$ measured at $t_0$ and $z_1$ measured at $t_1$...
0
votes
0answers
55 views

Updating biasses while backpropagation in all details

I just wabt to ask do i understand the usage and updating biasses while backpropagating and what values comes where. Let us see following network : with bach size 6, input size 2, hidden size 4 and ...
0
votes
0answers
126 views

How to back-propagate illegal actions for policy gradient learning

When training a AI RL agent to play a game there'll be situations where the AI cannot perform certain actions lest they violate the game rules. That's easy to handle, and I can set illegal actions to ...
0
votes
1answer
118 views

Training the generator in a GAN pair with back propagation

For the purposes of this question I am asking about training the generator, assume that training the discriminator is another topic. My understanding of generative adversarial networks is that you ...
0
votes
1answer
64 views

Back propagation on matrix of weights

I am trying to implement a Neural Network for binary classification using python and numpy only. My network structure is as follows: input features: 2 [1X2] matrix Hidden layer1: 5 neurons [2X5] ...