Questions tagged [backpropagation]

For questions about the back-propagation (aka "backprop", and often abbreviated as "BP") algorithm, which is used to compute the gradient of the objective function (e.g. the mean squared error) with respect to the parameters (or weights) of the neural network, when trained with gradient descent.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

NaN gradients while Training (but loss isn't NaN neither the computational graph is disconnected)

The reason I am asking this here is that I haven't found a bug in my code and maybe there isn't a bug at all (or maybe there is). I just want to validate the idea that I am trying to implement. Here ...
Ryukendo Dey's user avatar
1 vote
1 answer
102 views

Why in one article to calcule the delta uses the weights of current layer, while in another article uses the weights of neuron inputs of next layer?

Article 1: https://pyimagesearch.com/2021/05/06/backpropagation-from-scratch-with-python/ Article 2: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/ I was ...
will The J's user avatar
1 vote
0 answers
35 views

Neural network learns to mimic distribution of classes in dataset instead of using signal from input

I'm trying to implement example from a classic AI paper named "Learning representations by back-propagating errors" by Hinton et al. Example aims at training network able to predict third ...
Jan Grzybek's user avatar
0 votes
0 answers
29 views

How was the word2vec model trained?

Let's take the CBOW (continuous bag of words) model as the example. Suppose that, there are $c$ context words, each of which is a one-hot encoding vector. So the total number of elements of input ...
J. Doe's user avatar
  • 1
0 votes
1 answer
40 views

If I manually trained a multilayer percetron neural network, always following exactly the same steps meticulously, would I always get the same result?

Reference: https://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html If I trained a multilayer percetron neural network manually, following exactly the backpropagation steps described in the article, ...
will The J's user avatar
0 votes
2 answers
116 views

Does this article make use of the chain rule? And where?

References: Chain rule in Wikipedia: https://en.wikipedia.org/wiki/Chain_rule Chain rule in Towards Data Science: https://towardsdatascience.com/understanding-backpropagation-algorithm-7bb3aa2f95fd#:~...
will The J's user avatar
1 vote
1 answer
45 views

In multilayer perceptron neural networks, are the names "delta", "gradient" and "error" all the same thing? or not?

What is the difference between "delta", "gradient" and "error", are these names the same thing? I'm confused because someone once told me that both the names "delta&...
will The J's user avatar
1 vote
1 answer
64 views

Please, could someone help me understand if the backpropagation explanations in these two articles about calculate the error are equivalent?

I have a question about backpropagation, I'm a beginner, I'm studying the formulas to calculate the delta of neurons, there are several sources on the internet, which teach in different ways, so I'm ...
will The J's user avatar
0 votes
0 answers
17 views

Does the training time of a neural network increase more if we add a layer at the beginning or at the end?

Let's consider a fixed NN architecture, dataset and hardware. We add a layer, either at the beginning or at the end of the NN. In which case the training time will increase more? Intuitively, I ...
DeltaIV's user avatar
  • 194
1 vote
1 answer
359 views

what is the backpropagation formula to calculate delta and update weights?

I'm trying to study how backpropagation works step by step in a MultiLayer Perceptron neural network. I would really like to be able to understand how these calculations work. And I have a specific ...
will The J's user avatar
0 votes
0 answers
11 views

Bug in backpropagation algorithm

I've been debugging my backpropagation algorithm for nearly a week and I can't seem to find where I went wrong, yet my network doesn't seem to learn correctly. It learns XOR just fine when I use the ...
YamMan's user avatar
  • 1
0 votes
0 answers
20 views

Independent parameter update in backpropagation

When we calculate the gradient wrt to each paramters, we consider the other parameters remain constant, but the moment their is a change in any of the other parameters, shouldn't all the other changes ...
In progress...'s user avatar
0 votes
0 answers
18 views

Is it possible to combine SGD with an unsupervised learning approach effectively

Before I undertake quite a large project I would like to clarify whether my idea for training a multi-layer neural network will work. I plan to make an AI that can land a rocket from randomly ...
Gamaray's user avatar
0 votes
1 answer
85 views

Confusion over taking gradients in Variational Autoencoders (VAE)

I am confused as to when to hold certain parameters constant in a VAE. I will explain with a concrete example. We can write $\operatorname{ELBO}(\phi, \theta) = \mathbb{E}_{q_{\phi}(z)}\left[\log \...
Joel's user avatar
  • 23
0 votes
0 answers
39 views

neural network not training correctly

i'm trying to learn how a neural network works. i'm writing the neural network in C for handwritten digit recognition and training it on MNIST dataset. the neural network has an input layer with 28*28 ...
Faby's user avatar
  • 9
0 votes
1 answer
207 views

How to do backpropagation with argmax?

I am attempting to utilize two networks: a classifier and a linear network. Based on the output class of the first network, my goal is to retrieve the corresponding value from the linear network using ...
Subrat Prasad's user avatar
0 votes
1 answer
97 views

Does the output of LLM's affect their neural weights?

When an LLM creates an output, it seemingly has no way to check if its output was valid. Therefore it wouldn't be able to back-propagate any changes to the weights is used to create that output. ...
Seph Reed's user avatar
  • 103
1 vote
2 answers
104 views

Backpropagation with CrossEntropy and Softmax, HOW?

Let Zs be the input of the output layer (for example, Z1 is the input of the first neuron in the output layer), Os be the output of the output layer (which are actually the results of applying the ...
qazaq's user avatar
  • 11
1 vote
0 answers
32 views

Can gradient descent cause loss to increase in some situations?

Is a gradient descent step always supposed to decrease loss? I can think of a situation where it would seem that gradient descent would increase loss but maybe it I am misunderstanding a part of ...
Mike Levi's user avatar
1 vote
2 answers
56 views

How to apply backpropagation when one layer of the network is a call-only function (no gradient)?

I worked with Feed Forward Neural Network and VAE and understood backpropagation algorithm. Now I build a VAE network, one layer of it is a very complex vector-to-vector function $f(x)$ (a general '...
whitegreen's user avatar
0 votes
1 answer
55 views

About the requirement to compute the gradient at layer $l$

I'm trying to understand a line of my note. Let's say there is a simple feedforward neural network that has $N$ layers, and for a given layer $l$, it has weight $W^l$, and $g^l$ is the gradient to ...
NRain's user avatar
  • 111
1 vote
1 answer
121 views

Is there any reference about backpropagation of the Transformer's multi-head layer?

Is there any reference about backpropagation of the Transformer's multi-head layer or multi-head attention (MHA)? I have searched various journals but have not found one yet.
poglhar's user avatar
  • 23
1 vote
2 answers
39 views

Is there a resource that offers a detailed overview of the gradient flow?

Understanding the concept of "Gradient Flow" can be quite difficult as there is a lack of widely recognized and clearly defined resources that provide a comprehensive explanation. Although ...
v1998199904's user avatar
0 votes
1 answer
73 views

Is there a recommended resource that can provide a detailed overview of the gradient norm?

When it comes to the concept of "Gradient Norm," it can be challenging to find a widely recognized and clearly defined resource that offers a comprehensive explanation. While many search ...
StudentV's user avatar
2 votes
1 answer
373 views

How is the max function differentiable wrt multiple arguments?

I recently came across an answer on StackOverflow that mentioned the max function being differentiable with respect to its values. From my current understanding of ...
Peyman's user avatar
  • 554
10 votes
2 answers
806 views

Why use ReLU over Leaky ReLU?

From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive. With just this info to go off of,...
John Brown's user avatar
0 votes
0 answers
104 views

Computational overhead of "SCALING FORWARD GRADIENT WITH LOCAL LOSSES"

The paper "SCALING FORWARD GRADIENT WITH LOCAL LOSSES" discusses a new way of training deep neural networks called forward gradient learning. This method is different from the traditional ...
caesar's user avatar
  • 101
0 votes
0 answers
29 views

Is it possible to get recurrence through alternating forward and backward propagation?

This is a purely theoretical question, and I'm not trying to solve some real world problem with this, and I don't care about how ineffective this may be, as long as it is theoretically possible. For a ...
camel-cdr's user avatar
  • 101
0 votes
0 answers
48 views

Backpropagation of position-wise feedforward neural network

I have read a paper entitled "Attention is all you need" by Vaswani et al. (2017). This paper use the so-called position-wise feedforward neural network, where the input of this network is a ...
poglhar's user avatar
  • 23
0 votes
0 answers
103 views

Why Hadamard product appear in General backpropagation derivation?

I'm trying to understand the basic backpropagation of a neural network and stuck with this formula, which describes how you calculate the error vector for a layer $i$: $$\delta z^{[i]} = (w^{[i + 1]})^...
MathematicsBeginner's user avatar
3 votes
1 answer
101 views

What is the partial derivative $\frac{\partial y}{\partial x_1}$ in this neural network?

The answer is supposed to be -6, but I don't know how to get that. Also, in a NN, is that 2nd hidden layer possible, where the neurons are not dependent on all the neurons of the previous layer?
duanebobby's user avatar
1 vote
1 answer
110 views

How do policy gradients work?

If I understand it correctly from the following equation $$U(\theta)=\mathbb{E}_{\tau \sim P(\tau;\theta)}\left [ \sum_{t=0}^{H-1}R(s_t,u_t);\pi_{\theta} \right ]=\sum_{\tau}P(\tau;\theta)R(\tau)$$ ...
User's user avatar
  • 175
0 votes
1 answer
43 views

How would I use backpropagation with a changing cost function [closed]

I have a neural network that is being trained with a changing cost function. Could I use backpropagation at all? If yes, how would I do this?
922073's user avatar
  • 11
1 vote
2 answers
63 views

Why this single layer perceptron for the add operation not learning the correct weights?

This might be unnecessary but to learn the basics of neural networks, I am trying to create a single perceptron neural network to solve the adding operation of 2 inputs ...
EEAH's user avatar
  • 193
0 votes
0 answers
124 views

Can an LLM such as GPT learn from a sentence prediction that is close in meaning but drastically different in wording?

Consider this example: husband cheats on wife, AI predicts that wife gets angry. AI predicts that wife may either scream or throw things, with screaming a higher probability. Turns out in reality ...
Yan King Yin's user avatar
1 vote
1 answer
134 views

Is the re-parameterization trick necessary in the policy gradient method?

If we want to learn a stochastic policy with the policy gradient method, we have to sample from the distribution to get an action. Wouldn't this lead to the same issue that variational autoencoders ...
Sam's user avatar
  • 175
0 votes
1 answer
66 views

Seq2seq with RNNs, how is the training loop performed?

How do we train a seq2seq rnn training? We input a sentence that needs to be translated. We encode it sequentially. Then the first decoder outputs the first word with probabilities. We do a gradient ...
FluidMechanics Potential Flows's user avatar
1 vote
3 answers
88 views

Do Artificial Neural Network with non-linear activation only in the output layer follows linearity?

I am using a model with linear activation in the hidden layer and non-linear activation in the output layer. Could you please help to know whether such models exhibit linearity or not? The non-linear ...
Prabal Devkota's user avatar
0 votes
0 answers
10 views

Multiple network with one loss function, non-differentiable, is it okay with backpropagation?

Hello I'm trying to make the augmentation function which chooses to augmentate or not based on the contents of image. To make this, I'm thinking about using two different network. One is to classify ...
chan so's user avatar
0 votes
0 answers
42 views

How does the cross entropy loss function interact with the final layer of a neural network?

I am having trouble understanding how the result of categorical cross entropy loss can be used to calculate the gradient for all of the weights. The output of cross entropy function is the sum of all ...
Nick's user avatar
  • 1
2 votes
0 answers
62 views

Backpropagation with multiple output neurons but only one loss value

Suppose we have the following neural network (in reality it is a CNN with 60k parameters): This image, as well as the terminology used here, is borrowed from Matt Mazur As is visible, there are two ...
Value_Investor's user avatar
0 votes
0 answers
86 views

When should I think of using the Forward Forward algorithm?

Recently I read the paper named "The Forward-Forward Algorithm: Some Preliminary Investigations" and I was wondering what cases I should think of using it on. So according to the paper If we ...
HAMDI ABDERRAHMENE's user avatar
0 votes
1 answer
265 views

Single Layer Perceptron Backpropagation: How to compute affect of the net value on the output?

Assuming a single perceptron (see figure), I have found two versions of how to use backpropagation to update the weights. The perceptron is split in two, so we see the weighted sum on the left (the ...
HTH's user avatar
  • 1
0 votes
0 answers
41 views

Why differentiation of fourier operator is difficult?

I have a question when I read some papers about physics-informed neural networks. In the paper of physics-informed neural operator, they said "it is non-trivial to compute the derivatives for ...
徐宇霆's user avatar
0 votes
1 answer
72 views

Is it possible to reverse engineer out the loss based on weights update when data is unknown?

Assume the gradient updates (both $W_t$ and $W_{t+1}$) and learning rate are known while data $X$ is unknown, is it possible to deduce the loss $L$ used in backprop algorithm that gave rise to the ...
Sam's user avatar
  • 175
1 vote
2 answers
1k views

Why should one expect the backward pass to take twice as long as the forward pass?

I have seen it stated that, as a rule of thumb, a backward pass in a neural network should take about twice as long as the forward pass. Examples: From DeepSpeed's Flops Profiler docs, the profiler: ...
user26866's user avatar
  • 111
1 vote
2 answers
101 views

Back propagation activation function derivative [closed]

I am reading about backpropagation for fully connected neural networks and I found a very interesting article by Jeremy Jordan. It explains the process from start to finish. There is a section though ...
Kyriafinis Vasilis's user avatar
1 vote
0 answers
26 views

How to make my neural networks designs more robust

Whenever, I design a neural network to solve a novel problem (requires a novel loss function i.e. not image classification) it always ends up being hypersensitive to batch size and learning rate. ...
Tom Huntington's user avatar
0 votes
2 answers
233 views

How Does Convolution Backpropagation Work?

Assume in a convolutional layer's forward pass we have a $10\times10\times3$ image and five $3\times3\times3$ kernels, then $(10\times10\times3) *( 3\times3\times3\times5)$ has the output of ...
rkuang25's user avatar
1 vote
0 answers
71 views

Training loss is decreasing very slowly while learning MNIST database

I am developing my ANN from scratch which is supposed to classify MNIST database of handwritten digits (0-9). My feed-forward fully connected ANN has to be composed of: One input layer, with ...
Anna's user avatar
  • 11

1
2 3 4 5 6