# Tag Info

Accepted

### What is the time complexity for training a neural network using back-propagation?

I haven't seen an answer from a trusted source, but I'll try to answer this myself, with a simple example (with my current knowledge). In general, note that training an MLP using back-propagation is ...
Accepted

### What is "backprop"?

"Backprop" is the same as "backpropagation": it's just a shorter way to say it. It is sometimes abbreviated as "BP".
• 2,046

### What exactly is averaged when doing batch gradient descent?

Introduction First of all, it's completely normal that you are confused because nobody really explains this well and accurately enough. Here's my partial attempt to do that. So, this answer doesn't ...
• 35.6k

### What is "backprop"?

'Backprop' is short for 'backpropagation of error' in order to avoid confusion when using backpropagation term. Basically backpropagation refers to the method for computing the gradient of the case-...
• 10.2k
Accepted

### How to combine backpropagation in neural nets and reinforcement learning?

Gradient descent and back-propagation In deep learning, gradient descent (GD) and back-propagation (BP) are used to update the weights of the neural network. In reinforcement learning, one could map (...
Accepted

### How do evolutionary algorithms have advantages over the conventional backpropagation methods?

Unlike backpropagation, evolutionary algorithms do not require the objective function to be differential with respect to the parameters you aim to optimize. As a result, you can optimize "more things" ...
• 2,046
Accepted

### How to avoid falling into the "local minima" trap?

There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include: Probabalistically accepting worse solutions in the hope that this will ...
• 7,116
Accepted

### Do we know what the units of neural networks will do before we train them?

In reverse order to how you asked: all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal This actually happens if you ...
• 25k
Accepted

### Can non-differentiable layer be used in a neural network, if it's not learned?

It is not possible to backpropagate gradients through a layer with non-differentiable functions. However, the pooling layer function is differentiable*, and usually trivially so. For example: If an ...
• 25k
Accepted

### Is back-propagation applied for each data point or for a batch of data points?

Short answers Is back-propagation applied immediately after getting the output for each input or after getting the output for all inputs in a batch? You can perform back-propagation using (or after) ...
• 35.6k

### Why is it called back-propagation?

Why is it called back-propagation? I don't think there is anything special here! It's called back-propagation (BP) because, after the forward pass, you compute the partial derivative of the loss ...
• 35.6k
Accepted

### What do symmetric weights mean and how does it make backpropagation biologically implausible?

"Symmetric weights" means that the same weight value associated to a pair of nodes must be used during the forwards and backwards steps. The reason it makes back propagation biologically ...
• 4,593

### How do evolutionary algorithms have advantages over the conventional backpropagation methods?

Further to Franck's answer, there may be better optima (even global optima) that exist in the opposite direction to the gradient (which may be in the direction of some local optima). Evolutionary ...
• 712

### Why not teach to a NN not only what is true, but also what is not true?

Yes this is done routinely. For example this is how the YOLO object detection and classifier system works, to give a real-world for example. In YOLO, the "non-object" classification is "background" i....
• 25k

### What is the actual learning algorithm: back-propagation or gradient descent?

You can run gradient descent without back propagation, in some cases: Simple structures such as linear or logistic regression, where the gradients can be calculated directly from the inputs and cost ...
• 25k

### What is the relation between back-propagation and reinforcement learning?

Backpropagation is a subroutine often used when training Artificial Neural Networks with a Gradient Descent learning algorithm. Gradient Descent requires the computation of the error gradient, i.e. ...

### Are these two versions of back-propagation equivalent?

The two examples present essentially the same operation: In both cases, the network is trained with gradient descent using the backpropagated squared error computed at the output. Both examples use ...
• 933
Accepted

### Finding an optimum back propagation algorithm

When you are training a neural network, you use an algorithm called back propagation. This algorithm uses partial derivatives to determine the optimal values for weights. Partial derivatives are a ...

### Is the mean-squared error always convex in the context of neural networks?

Answer in short: MSE is convex on its input and parameters by itself. But on an arbitrary neural network it is not always convex due to the presence of non-linearities in the form of activation ...
• 562

### What is the time complexity for training a neural network using back-propagation?

For the evaluation of a single pattern, you need to process all weights and all neurons. Given that every neuron has at least one weight, we can ignore them, and have $\mathcal{O}(w)$ where $w$ is the ...
• 523

### What is the actual learning algorithm: back-propagation or gradient descent?

Gradient descent (GD) is an optimisation algorithm, that is, it is used to find a (local) minimum of a multi-variable and differentiable function $f$. GD is an iterative and numerical optimisation ...
• 35.6k
Accepted

### Are filters fixed or learned?

What are filters in image processing? In the context of image processing (and, in general, signal processing), the kernels (also known as filters) are used to perform some specific operation on the ...
• 35.6k

### What is the purpose of the batch size in neural networks?

tl;dr: A batch size is the number of samples a network sees before updating its gradients. This number can range from a single sample to the whole training set. Empirically, there is a sweet spot in ...
• 3,113
Accepted

### How can I use a Hidden Markov Model to recognize images?

You wouldn't, normally. A HMM is used to model sequences of observations, and it would not make sense to use it for image recognition. Unless they are sequential, such as strokes in handwriting. HMMs ...
• 5,187
Accepted

### How to test if my implementation of back propagation neural Network is correct

Actually the implementation was correct, The source of the problem that causes a big error and really slow learning was the architecture of the neural network it self, the ANN has 7 hidden layers ...
Accepted

### How do I know if my backpropagation is implemented correctly?

Don't feel too bad for having gotten it slightly wrong because backpropagation is notoriously difficult to implement [1]. There is a technique called gradient checking, which you can implement to test ...

### Does training happen during NEAT?

NEAT uses genetic algorithms both to search for improved connection weights and for improved architectures. Whilst it is possible to train a NEAT-generated neural network using backpropagation of ...
• 25k
Accepted

### Could error surface shape be useful to detect which local minima is better for generalization?

In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations: what the author of the question provides us with is the Loss ...