# Tag Info

Accepted

### What are the implications of the "No Free Lunch" theorem for machine learning?

This is a really common reaction after first encountering the No Free Lunch theorems (NFLs). The one for machine learning is especially unintuitive, because it flies in the face of everything that's ...
• 9,327
Accepted

### Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. I ...
• 32.8k
Accepted

### Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch

I see no reason why decaying learning rates should create the kinds of jumps in losses that you are observing. It should "slow down" how quickly you "move", which in the case of a loss that otherwise ...
• 10.4k
Accepted

### Why is the learning rate generally beneath 1?

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition $$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$ where $a_t$ is the learning rate at iteration $t$, ...
• 41k

### Can artificial intelligence be thought of as optimization?

A good answer to this question depends on what you want to use the labels for. When I think about "optimization," I think about a solution space and a cost function; that is, there are many possible ...
• 4,272
Accepted

### How to avoid falling into the "local minima" trap?

There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include: Probabalistically accepting worse solutions in the hope that this will ...
• 7,276
Accepted

$g(x) = x^2$ is indeed a parabola and thus has just one optimum. However, the $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - f(x_i))^2$, where $\boldsymbol{x}$ are the inputs, $\... • 41k 9 votes Accepted ### What is the difference between reinforcement learning and evolutionary algorithms? Evolutionary algorithms (EAs) are a family of algorithms inspired by the biological evolution that can be used to solve (constrained or not) optimization problems where the function that needs to be ... • 41k 9 votes ### Why is gradient descent used over the conjugate gradient method? When dealing with optimization problems, a fundamental distinction is whether the objective is a (deterministic) function, or an expectation of some function. I will refer to these cases as the ... • 1,291 8 votes Accepted ### What are hyper-heuristics, and how are they different from meta-heuristics? TL:DR: Hyper-heuristics are metaheuristics, suited for solving the same kind of optimization problems, but (in principle) affording a "rapid prototyping" approach for non-expert practitioners. In ... • 7,276 8 votes Accepted ### What is an objective function? The "objective function" is the function that you want to minimise or maximise in your problem. The expression "objective function" is used in several different contexts (e.g. machine learning or ... • 41k 7 votes ### What is the actual learning algorithm: back-propagation or gradient descent? You can run gradient descent without back propagation, in some cases: Simple structures such as linear or logistic regression, where the gradients can be calculated directly from the inputs and cost ... • 32.8k 7 votes Accepted ### Why is the perceptron criterion function differentiable?$\max(-y_i(w x_i), 0)$is not partial derivable respect$w$if$w x_i=0$. Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval ... • 1,293 6 votes Accepted ### If Deep Learning is non convex, then why do use a convex loss function? Well, you are definitely mixing two different things. Here are those bits: The function that deep learning approximates is basically a function that best fits the INPUT DATA points. You should not ... • 186 6 votes Accepted ### What are the limitations of the hill climbing algorithm and how to overcome them? As @nbro has already said that Hill Climbing is a family of local search algorithms. So, when you said Hill Climbing in the question I have assumed you are talking about the standard hill climbing. ... • 2,023 6 votes Accepted ### When should we use algorithms like Adam as opposed to SGD? Empirically, I observed that algorithms like Adam and RMSProp tended to give me a final higher performance (in my case, the accuracy) on (the validation dataset) with respect to SGD. However, I also ... • 41k 6 votes Accepted ### Why is second-order backpropagation useful? Second-order optimization algorithms like Hessian optimization have more information on the curvature of the loss function, so converge much, much faster than first-order optimization algorithms like ... • 1,650 6 votes Accepted ### When should you not use the bias in a layer? The most usual case of bias=False is in layers before/after Batch Normalization with no activators in between. The BatchNorm layer will re-center the data anyway, ... • 2,544 5 votes Accepted ### What AI technique should I use to assign a person to a task? What you have could be well described as a Task Allocation problem, which is studied as part of the planning subfield of AI. Chapters 10 & 11 of Russell & Norvig provide a good overview of ... • 9,327 5 votes ### Can artificial intelligence be thought of as optimization? I can offer two (at first sight, conflicting) perspectives on this: Firstly: If the letter string 'abc' becomes 'abd' what would "doing the same thing" to 'ijk' look like? This is just one example ... • 7,276 5 votes ### What is the actual learning algorithm: back-propagation or gradient descent? Gradient descent (GD) is an optimisation algorithm, that is, it is used to find a (local) minimum of a multi-variable and differentiable function$f$. GD is an iterative and numerical optimisation ... • 41k 5 votes ### What are the limitations of the hill climbing algorithm and how to overcome them? Hill climbing is not an algorithm, but a family of "local search" algorithms. Specific algorithms which fall into the category of "hill climbing" algorithms are 2-opt, 3-opt, 2.5-opt, 4-opt, or, in ... • 41k 5 votes ### In deep learning, is it possible to use discontinuous activation functions? Even the first artificial neural network - Rosenblatt's perceptron [1] had a discontinuous activation function. That network is in introductory chapters of many textbooks about AI. For example, ... 4 votes ### What are the techniques for detecting and preventing overfitting? Usually you keep track of training loss and validation loss and apply proper regularization technique (such as L1, L2, dropout, DropConnect, etc.). The more interesting technique is to observe your ... • 341 4 votes ### When should I use simulated annealing as opposed to a genetic algorithm? Simulated Annealing vs genetic algorithm? Simulated annealing is a materials science analogy and involves the introduction of noise to avoid search failure due to local minima. See images below. To ... • 7,513 4 votes Accepted ### Could error surface shape be useful to detect which local minima is better for generalization? In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations: what the author of the question provides us with is the Loss ... 4 votes Accepted ### How are these equations of SGD with momentum equivalent? The first two equations are equivalent. The last equation can be equivalent if you scale$\alpha$appropriately. Equation 1 Consider the equation from the Stanford slide:$\$ v_{t}=\rho v_{t-1}+\nabla ...
• 1,650

The fundamental issue is that one doesn't really want to find an optimum of one's optimization problem. We are really interested in generalization - not optimality. And we still poorly understand how ...
• 2,544
Accepted

### Why isn’t this formula used at all?

Automatic Differentiation (AD) is a technique used to compute the derivatives of functions with respect to their inputs applying chain rules, and the famous backpropagation algo used in most ANNs is a ...
• 2,821