Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [backpropagation]

For questions related to the technique of backpropagation, whereby the loss, error, or correction signal calculated at the output of an artificial network output is fed back to the parameters in each layer of the network until the network's behavior converges to a training state within the required accuracy and reliability.

30 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
2answers
59 views

Why do very deep non resnet architectures perform worse compared to shallower ones for the same iteration? Shouldn't they just train slower?

My understanding of the vanishing gradient problem in deep networks is that as backprop progresses through the layers the gradients become small, and thus training progresses slower. I'm having a hard ...
5
votes
0answers
123 views

How to design 4D Deep Recurrent Neural Networks using Tensorflow?

I want to design a simple model that predicts the movement of coordinates with RNNs. In a typical three-dimensional LSTM model, one feature is encoded as one hot encoding, and the ...
4
votes
1answer
75 views

Backpropagation equation for a variant on the usual Linear Neuron architecture

Recently I encountered a variant on the normal linear neural layer architecture: Instead of $Z = XW + B$, we now have $Z = (X-A)W + B$. So we have a 'pre-bias' $A$ that affects the activation of the ...
3
votes
0answers
498 views

How to calculate gradient of filter in convolution network

I have similar architecture like in image:CNN. I don't understand how to calculate gradient of filter F. I found these equations(source): Gradient and delta, where first equation calculate gradient ...
2
votes
0answers
11 views

Calculation of Neural network biases in backpropagation

While learning neural networks I've found a basic Python working example to play with. It has 3 input nodes, 4 nodes in a hidden layer, 1 output node. 5 data sets for training. The initial code is ...
2
votes
0answers
22 views

How does a Bidirectional RNN work?

Could it be possible to reach a similar output via feeding a unidirectional network with the original data and the data played backwards?
2
votes
2answers
76 views

Which neuron represents which part of the input?

In a neural network, each neuron represents some part of the input. For example, in the case of a MNIST digit, consider the stem of the number 9. Each neuron in the NN represents some part of this ...
2
votes
0answers
96 views

Update of weights in Recurrent Neural Network through back propagation

How does Recurrent Neural Network updates its weights and bias through backpropagation? Is time taken into account while updating the weights of a RNN using Backpropagation through time(BPTT)?"
2
votes
2answers
485 views

How to perform neural network with output constraint?

Imagine a "simple" feedforward, fully connected neural network, with some input size, some number of hidden layers, and some # of neurons....etc BUT with a fixed number of output size (that is saying, ...
2
votes
0answers
107 views

Data prepared to linear regression. Can I use it with backpropagation?

I'm studying a Master's Degree in Artificial Intelligence and I need to learn how to use the Java Neural Network Simulator, JavaNNS, program. In one practice I have to build a neural network to use ...
2
votes
0answers
46 views

Recommendations on which architecture to use to guess appointment

I'm currently developping an application which allows psychologists to manage their schedule and budget. As a proof of concept, I would like to create an intelligent appointment service. There can be ...
2
votes
1answer
81 views

Backpropagation in Decoupled Neural Interfaces

I am attempting to create a fully decoupled feed-forward neural network by using decoupled neural interfaces as explained in the paper (https://arxiv.org/abs/1608.05343). As in the paper, the DNI is ...
1
vote
0answers
4 views

yolo output and how to define labels for backpropogation on it

I want to build the yolo architecture in keras but can't understand the basic idea behind the training of the yolo, like how to define the labels for whether there is no object there what we have to ...
1
vote
0answers
30 views

Understanding the partial derivative with respect to the weight matrix and bias

Say we have the layer $X W + b = Y$. I want to get $\frac{dL}{dW}$ and we assume I have $\frac{dL}{dY}$. So all I need is to find $\frac{dY}{dW}$. I know that it should be $X^T\frac{dL}{dY}$ but don'...
1
vote
0answers
28 views

How does a single neuron in hidden layer affect training accuracy

I'm currently a student learning about AI Networks. I've came across a statement in one of my Professor's books that a FFBP (Feed-Forward Back-Propagation) Neural Network with a single hidden layer ...
1
vote
0answers
49 views

How to train and update weights of filters

I have some problems with training CNN :( For example: Input 6x6x3, 1 core 3x3x3, output = 4x4x1 => pool: 2x2x1 By backpropagation I calculated deltas for output. This tutor and other tutors are ...
1
vote
0answers
48 views

Difficulties to implement the layer-wise relevance propagation in MATLAB

I'm having serious issues with the implementation of the LRP algorithm for neural networks in MATLAB. The challenge is to implement the equations correctly. I'm trying to implement the deep-Taylor $\...
1
vote
0answers
41 views

How are the weights between the input and hidden layer updated in a 3 layer neural network?

Consider a feed-forward neural network with one hidden layer. How are the weights between the input and hidden layer updated, after the weights between the hidden layer and output layer are updated?
1
vote
0answers
28 views

Use of backpropagation for weight updates in a combination of 2 neural networks

Every neural network updates its weights through back-propagation. How is back-propagation used for updating weights in a combination of 2 or more neural networks (e.g.:CNN-LSTM, GAN-CNN, etc.). For ...
1
vote
0answers
47 views

Using features extracted from a CNN as convolutional filter

I'm a bit confused about this. Assume I have a CNN network with two branches: Top Bottom The top branch outputs a feature vector of shape 1x1x1x10 (batch, h, w, c) The bottom branch outputs a ...
1
vote
0answers
29 views

Has anyone investigated iteration awareness beyond RNN and LSTM?

This question considers the convergence of an artificial networks (MLPs, RNNs, LSTM nets, CNNs) over time or over the course of epochs made up of iterations through training examples. In this ...
1
vote
0answers
292 views

How does backpropagation work on a custom loss function whose components have magnitudes of different orders?

I want to use a custom loss function which is a weighted combination of l1 and DSSIM losses. The DSSIM loss is limited between 0 and 0.5 where as the l1 loss can be orders of magnitude greater and is ...
1
vote
0answers
85 views

When do you back-propagate errors through a Neural Network when using TD Lambda

I have a Neural Network that I'm want to use to self-play Connect Four. The neural network receives the board state and is to provide an estimate of the states desirability. I would then, for each ...
1
vote
1answer
330 views

How to find partial derivative of softmax w.r.t logits in python

i have trouble implementing back propogation for multi class classification of CIFAR10 dataset My neural network has 2 layers forward propagation X -> L1 -> L2 weights W are initialized as random ...
0
votes
0answers
33 views

Not able to properly tune Neural Network via Back Propagation properly

I have a custom code Neural Network(not using keras or any package...Trying to learn the essence of Neural Network from scratch)... Code can be found here I have the per iteration training output(<...
0
votes
0answers
35 views

Confused about NeuralODE

I am a bit confused about NeuralODE and I want to make sure that what I understood so far is correct. Assume we have (for simplicity) 2 data points $z_0$ measured at $t_0$ and $z_1$ measured at $t_1$...
0
votes
0answers
51 views

Updating biasses while backpropagation in all details

I just wabt to ask do i understand the usage and updating biasses while backpropagating and what values comes where. Let us see following network : with bach size 6, input size 2, hidden size 4 and ...
0
votes
0answers
106 views

How to back-propagate illegal actions for policy gradient learning

When training a AI RL agent to play a game there'll be situations where the AI cannot perform certain actions lest they violate the game rules. That's easy to handle, and I can set illegal actions to ...
0
votes
1answer
66 views

Training the generator in a GAN pair with back propagation

For the purposes of this question I am asking about training the generator, assume that training the discriminator is another topic. My understanding of generative adversarial networks is that you ...
0
votes
1answer
47 views

Back propagation on matrix of weights

I am trying to implement a Neural Network for binary classification using python and numpy only. My network structure is as follows: input features: 2 [1X2] matrix Hidden layer1: 5 neurons [2X5] ...