26 votes
Accepted

What is the time complexity for training a neural network using back-propagation?

I haven't seen an answer from a trusted source, but I'll try to answer this myself, with a simple example (with my current knowledge). In general, note that training an MLP using back-propagation is ...
user avatar
15 votes
Accepted

What is "backprop"?

"Backprop" is the same as "backpropagation": it's just a shorter way to say it. It is sometimes abbreviated as "BP".
user avatar
9 votes

What is "backprop"?

'Backprop' is short for 'backpropagation of error' in order to avoid confusion when using backpropagation term. Basically backpropagation refers to the method for computing the gradient of the case-...
user avatar
  • 10k
8 votes
Accepted

How to combine backpropagation in neural nets and reinforcement learning?

Gradient descent and back-propagation In deep learning, gradient descent (GD) and back-propagation (BP) are used to update the weights of the neural network. In reinforcement learning, one could map (...
8 votes
Accepted

How to avoid falling into the "local minima" trap?

There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include: Probabalistically accepting worse solutions in the hope that this will ...
user avatar
8 votes
Accepted

Do we know what the units of neural networks will do before we train them?

In reverse order to how you asked: all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal This actually happens if you ...
user avatar
  • 23.8k
8 votes

What exactly is averaged when doing batch gradient descent?

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
user avatar
  • 33.8k
8 votes
Accepted

What do symmetric weights mean and how does it make backpropagation biologically implausible?

"Symmetric weights" means that the same weight value associated to a pair of nodes must be used during the forwards and backwards steps. The reason it makes back propagation biologically ...
user avatar
7 votes
Accepted

How do evolutionary algorithms have advantages over the conventional backpropagation methods?

Unlike backpropagation, evolutionary algorithms do not require the objective function to be differential with respect to the parameters you aim to optimize. As a result, you can optimize "more things" ...
user avatar
7 votes

Why not teach to a NN not only what is true, but also what is not true?

Yes this is done routinely. For example this is how the YOLO object detection and classifier system works, to give a real-world for example. In YOLO, the "non-object" classification is "background" i....
user avatar
  • 23.8k
7 votes
Accepted

Can non-differentiable layer be used in a neural network, if it's not learned?

It is not possible to backpropagate gradients through a layer with non-differentiable functions. However, the pooling layer function is differentiable*, and usually trivially so. For example: If an ...
user avatar
  • 23.8k
7 votes

What is the actual learning algorithm: back-propagation or gradient descent?

You can run gradient descent without back propagation, in some cases: Simple structures such as linear or logistic regression, where the gradients can be calculated directly from the inputs and cost ...
user avatar
  • 23.8k
7 votes
Accepted

Is back-propagation applied for each data point or for a batch of data points?

Short answers Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch? You can perform back-propagation using (or after) ...
user avatar
  • 33.8k
7 votes

Why is it called back-propagation?

Why is it called back-propagation? I don't think there is anything special here! It's called back-propagation (BP) because, after the forward pass, you compute the partial derivative of the loss ...
user avatar
  • 33.8k
6 votes

How do evolutionary algorithms have advantages over the conventional backpropagation methods?

Further to Franck's answer, there may be better optima (even global optima) that exist in the opposite direction to the gradient (which may be in the direction of some local optima). Evolutionary ...
user avatar
6 votes

What is the relation between back-propagation and reinforcement learning?

Backpropagation is a subroutine often used when training Artificial Neural Networks with a Gradient Descent learning algorithm. Gradient Descent requires the computation of the error gradient, i.e. ...
user avatar
5 votes
Accepted

Finding an optimum back propagation algorithm

When you are training a neural network, you use an algorithm called back propagation. This algorithm uses partial derivatives to determine the optimal values for weights. Partial derivatives are a ...
user avatar
5 votes

Are these two versions of back-propagation equivalent?

The two examples present essentially the same operation: In both cases, the network is trained with gradient descent using the backpropagated squared error computed at the output. Both examples use ...
user avatar
5 votes

What is the time complexity for training a neural network using back-propagation?

For the evaluation of a single pattern, you need to process all weights and all neurons. Given that every neuron has at least one weight, we can ignore them, and have $\mathcal{O}(w)$ where $w$ is the ...
user avatar
5 votes

What is the actual learning algorithm: back-propagation or gradient descent?

Gradient descent (GD) is an optimisation algorithm, that is, it is used to find a (local) minimum of a multi-variable and differentiable function $f$. GD is an iterative and numerical optimisation ...
user avatar
  • 33.8k
5 votes

What is the purpose of the batch size in neural networks?

tl;dr: A batch size is the number of samples a network sees before updating its gradients. This number can range from a single sample to the whole training set. Empirically, there is a sweet spot in ...
user avatar
  • 3,093
5 votes
Accepted

How can I use a Hidden Markov Model to recognize images?

You wouldn't, normally. A HMM is used to model sequences of observations, and it would not make sense to use it for image recognition. Unless they are sequential, such as strokes in handwriting. HMMs ...
user avatar
  • 5,062
4 votes
Accepted

How to test if my implementation of back propagation neural Network is correct

Actually the implementation was correct, The source of the problem that causes a big error and really slow learning was the architecture of the neural network it self, the ANN has 7 hidden layers ...
user avatar
4 votes

Is the mean-squared error always convex in the context of neural networks?

Answer in short: MSE is convex on its input and parameters by itself. But on an arbitrary neural network it is not always convex due to the presence of non-linearities in the form of activation ...
user avatar
  • 532
4 votes
Accepted

How do I know if my backpropagation is implemented correctly?

Don't feel too bad for having gotten it slightly wrong because backpropagation is notoriously difficult to implement [1]. There is a technique called gradient checking, which you can implement to test ...
user avatar
4 votes
Accepted

Could error surface shape be useful to detect which local minima is better for generalization?

In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations: what the author of the question provides us with is the Loss ...
user avatar
4 votes
Accepted

Are filters fixed or learned?

What are filters in image processing? In the context of image processing (and, in general, signal processing), the kernels (also known as filters) are used to perform some specific operation on the ...
user avatar
  • 33.8k
4 votes
Accepted

How can the derivative of a neural network be calculated, given no mathematical expression?

I think what you mean to ask is how can differentiation occur when there's no obvious neural network function to differentiate? Don't worry - lots of people get confused about this, because it seems ...
user avatar
  • 156
4 votes

In deep learning, is it possible to use discontinuous activation functions?

Even the first artificial neural network - Rosenblatt's perceptron [1] had a discontinuous activation function. That network is in introductory chapters of many textbooks about AI. For example, ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible