Skip to main content
14 votes
Accepted

Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. I ...
Neil Slater's user avatar
  • 32.7k
14 votes
Accepted

What are the implications of the "No Free Lunch" theorem for machine learning?

This is a really common reaction after first encountering the No Free Lunch theorems (NFLs). The one for machine learning is especially unintuitive, because it flies in the face of everything that's ...
John Doucette's user avatar
11 votes
Accepted

Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch

I see no reason why decaying learning rates should create the kinds of jumps in losses that you are observing. It should "slow down" how quickly you "move", which in the case of a loss that otherwise ...
Dennis Soemers's user avatar
  • 10.3k
11 votes
Accepted

Why is the learning rate generally beneath 1?

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition $$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$ where $a_t$ is the learning rate at iteration $t$, ...
nbro's user avatar
  • 40.9k
10 votes

Can artificial intelligence be thought of as optimization?

A good answer to this question depends on what you want to use the labels for. When I think about "optimization," I think about a solution space and a cost function; that is, there are many possible ...
Matthew Gray's user avatar
  • 4,272
9 votes
Accepted

How is it possible that the MSE used to train neural networks with gradient descent has multiple local minima?

$g(x) = x^2$ is indeed a parabola and thus has just one optimum. However, the $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - f(x_i))^2$, where $\boldsymbol{x}$ are the inputs, $\...
nbro's user avatar
  • 40.9k
9 votes
Accepted

What is the difference between reinforcement learning and evolutionary algorithms?

Evolutionary algorithms (EAs) are a family of algorithms inspired by the biological evolution that can be used to solve (constrained or not) optimization problems where the function that needs to be ...
nbro's user avatar
  • 40.9k
9 votes

Why is gradient descent used over the conjugate gradient method?

When dealing with optimization problems, a fundamental distinction is whether the objective is a (deterministic) function, or an expectation of some function. I will refer to these cases as the ...
Taw's user avatar
  • 1,281
8 votes
Accepted

What are hyper-heuristics, and how are they different from meta-heuristics?

TL:DR: Hyper-heuristics are metaheuristics, suited for solving the same kind of optimization problems, but (in principle) affording a "rapid prototyping" approach for non-expert practitioners. In ...
NietzscheanAI's user avatar
8 votes
Accepted

How to avoid falling into the "local minima" trap?

There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include: Probabalistically accepting worse solutions in the hope that this will ...
NietzscheanAI's user avatar
8 votes
Accepted

What is an objective function?

The "objective function" is the function that you want to minimise or maximise in your problem. The expression "objective function" is used in several different contexts (e.g. machine learning or ...
nbro's user avatar
  • 40.9k
7 votes

What is the actual learning algorithm: back-propagation or gradient descent?

You can run gradient descent without back propagation, in some cases: Simple structures such as linear or logistic regression, where the gradients can be calculated directly from the inputs and cost ...
Neil Slater's user avatar
  • 32.7k
7 votes
Accepted

Why is the perceptron criterion function differentiable?

$\max(-y_i(w x_i), 0)$ is not partial derivable respect $w$ if $w x_i=0$. Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval ...
pasaba por aqui's user avatar
6 votes
Accepted

If Deep Learning is non convex, then why do use a convex loss function?

Well, you are definitely mixing two different things. Here are those bits: The function that deep learning approximates is basically a function that best fits the INPUT DATA points. You should not ...
mausamsion's user avatar
6 votes
Accepted

What are the limitations of the hill climbing algorithm and how to overcome them?

As @nbro has already said that Hill Climbing is a family of local search algorithms. So, when you said Hill Climbing in the question I have assumed you are talking about the standard hill climbing. ...
Ugnes's user avatar
  • 2,023
6 votes
Accepted

When should we use algorithms like Adam as opposed to SGD?

Empirically, I observed that algorithms like Adam and RMSProp tended to give me a final higher performance (in my case, the accuracy) on (the validation dataset) with respect to SGD. However, I also ...
nbro's user avatar
  • 40.9k
6 votes
Accepted

Why is second-order backpropagation useful?

Second-order optimization algorithms like Hessian optimization have more information on the curvature of the loss function, so converge much, much faster than first-order optimization algorithms like ...
user3667125's user avatar
  • 1,600
6 votes
Accepted

When should you not use the bias in a layer?

The most usual case of bias=False is in layers before/after Batch Normalization with no activators in between. The BatchNorm layer will re-center the data anyway, ...
Kostya's user avatar
  • 2,534
5 votes
Accepted

What AI technique should I use to assign a person to a task?

What you have could be well described as a Task Allocation problem, which is studied as part of the planning subfield of AI. Chapters 10 & 11 of Russell & Norvig provide a good overview of ...
John Doucette's user avatar
5 votes

Can artificial intelligence be thought of as optimization?

I can offer two (at first sight, conflicting) perspectives on this: Firstly: If the letter string 'abc' becomes 'abd' what would "doing the same thing" to 'ijk' look like? This is just one example ...
NietzscheanAI's user avatar
5 votes

What is the actual learning algorithm: back-propagation or gradient descent?

Gradient descent (GD) is an optimisation algorithm, that is, it is used to find a (local) minimum of a multi-variable and differentiable function $f$. GD is an iterative and numerical optimisation ...
nbro's user avatar
  • 40.9k
5 votes

What are the limitations of the hill climbing algorithm and how to overcome them?

Hill climbing is not an algorithm, but a family of "local search" algorithms. Specific algorithms which fall into the category of "hill climbing" algorithms are 2-opt, 3-opt, 2.5-opt, 4-opt, or, in ...
nbro's user avatar
  • 40.9k
5 votes

In deep learning, is it possible to use discontinuous activation functions?

Even the first artificial neural network - Rosenblatt's perceptron [1] had a discontinuous activation function. That network is in introductory chapters of many textbooks about AI. For example, ...
Vladislav Gladkikh's user avatar
4 votes

What are the techniques for detecting and preventing overfitting?

Usually you keep track of training loss and validation loss and apply proper regularization technique (such as L1, L2, dropout, DropConnect, etc.). The more interesting technique is to observe your ...
FunkyKowal's user avatar
4 votes

When should I use simulated annealing as opposed to a genetic algorithm?

Simulated Annealing vs genetic algorithm? Simulated annealing is a materials science analogy and involves the introduction of noise to avoid search failure due to local minima. See images below. To ...
Douglas Daseeco's user avatar
4 votes
Accepted

Could error surface shape be useful to detect which local minima is better for generalization?

In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations: what the author of the question provides us with is the Loss ...
Nicola Bernini's user avatar
4 votes
Accepted

How are these equations of SGD with momentum equivalent?

The first two equations are equivalent. The last equation can be equivalent if you scale $\alpha$ appropriately. Equation 1 Consider the equation from the Stanford slide: $$ v_{t}=\rho v_{t-1}+\nabla ...
user3667125's user avatar
  • 1,600
4 votes

Why is gradient descent used over the conjugate gradient method?

The fundamental issue is that one doesn't really want to find an optimum of one's optimization problem. We are really interested in generalization - not optimality. And we still poorly understand how ...
Kostya's user avatar
  • 2,534
3 votes
Accepted

Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?

Has this been done? Difficult to prove a negative, but I suspect although plenty of research has been done into finding ideal learning rate values (the need for learning rate at all is an annoyance), ...
Neil Slater's user avatar
  • 32.7k
3 votes
Accepted

Can I compute the fitness of an agent based on a low number of runs of the game?

You can probably get away with a relatively low X for two reasons: The Central Limit Theorem. This tells us that the accuracy in the estimate of an agent's fitness will improve as the square root of ...
John Doucette's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible