14
votes
Accepted
Why should the number of neurons in a hidden layer be a power of 2?
I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster.
I ...
- 30.3k
14
votes
Accepted
What are the implications of the "No Free Lunch" theorem for machine learning?
This is a really common reaction after first encountering the No Free Lunch theorems (NFLs). The one for machine learning is especially unintuitive, because it flies in the face of everything that's ...
- 9,207
11
votes
Accepted
Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch
I see no reason why decaying learning rates should create the kinds of jumps in losses that you are observing. It should "slow down" how quickly you "move", which in the case of a loss that otherwise ...
- 10.1k
11
votes
Accepted
Why is the learning rate generally beneath 1?
If the learning rate is greater than or equal to $1$ the Robbins-Monro condition
$$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$
where $a_t$ is the learning rate at iteration $t$, ...
- 39.6k
10
votes
Can artificial intelligence be thought of as optimization?
A good answer to this question depends on what you want to use the labels for.
When I think about "optimization," I think about a solution space and a cost function; that is, there are many possible ...
- 4,262
9
votes
Accepted
How is it possible that the MSE used to train neural networks with gradient descent has multiple local minima?
$g(x) = x^2$ is indeed a parabola and thus has just one optimum.
However, the $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - f(x_i))^2$, where $\boldsymbol{x}$ are the inputs, $\...
- 39.6k
9
votes
Accepted
What is the difference between reinforcement learning and evolutionary algorithms?
Evolutionary algorithms (EAs) are a family of algorithms inspired by the biological evolution that can be used to solve (constrained or not) optimization problems where the function that needs to be ...
- 39.6k
8
votes
Accepted
What are hyper-heuristics, and how are they different from meta-heuristics?
TL:DR: Hyper-heuristics are metaheuristics, suited for solving the same kind of optimization problems, but (in principle) affording a "rapid prototyping" approach for non-expert practitioners. In ...
- 7,226
8
votes
Accepted
How to avoid falling into the "local minima" trap?
There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include:
Probabalistically accepting worse solutions in the hope that this
will ...
- 7,226
8
votes
Accepted
What is an objective function?
The "objective function" is the function that you want to minimise or maximise in your problem.
The expression "objective function" is used in several different contexts (e.g. machine learning or ...
- 39.6k
7
votes
What is the actual learning algorithm: back-propagation or gradient descent?
You can run gradient descent without back propagation, in some cases:
Simple structures such as linear or logistic regression, where the gradients can be calculated directly from the inputs and cost ...
- 30.3k
7
votes
Accepted
Why is the perceptron criterion function differentiable?
$\max(-y_i(w x_i), 0)$ is not partial derivable respect $w$ if $w x_i=0$.
Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval ...
- 1,283
7
votes
Why is gradient descent used over the conjugate gradient method?
When dealing with optimization problems, a fundamental distinction is whether the objective is a (deterministic) function, or an expectation of some function. I will refer to these cases as the ...
- 1,201
6
votes
Accepted
If Deep Learning is non convex, then why do use a convex loss function?
Well, you are definitely mixing two different things. Here are those bits:
The function that deep learning approximates is basically a function that best fits the INPUT DATA points. You should not ...
- 186
6
votes
Accepted
What are the limitations of the hill climbing algorithm and how to overcome them?
As @nbro has already said that Hill Climbing is a family of local search algorithms. So, when you said Hill Climbing in the question I have assumed you are talking about the standard hill climbing. ...
- 2,023
6
votes
Accepted
When should we use algorithms like Adam as opposed to SGD?
Empirically, I observed that algorithms like Adam and RMSProp tended to give me a final higher performance (in my case, the accuracy) on (the validation dataset) with respect to SGD. However, I also ...
- 39.6k
5
votes
Accepted
What AI technique should I use to assign a person to a task?
What you have could be well described as a Task Allocation problem, which is studied as part of the planning subfield of AI. Chapters 10 & 11 of Russell & Norvig provide a good overview of ...
- 9,207
5
votes
Can artificial intelligence be thought of as optimization?
I can offer two (at first sight, conflicting) perspectives on this:
Firstly:
If the letter string 'abc' becomes 'abd' what would "doing the same thing" to 'ijk' look like?
This is just one example ...
- 7,226
5
votes
What is the actual learning algorithm: back-propagation or gradient descent?
Gradient descent (GD) is an optimisation algorithm, that is, it is used to find a (local) minimum of a multi-variable and differentiable function $f$. GD is an iterative and numerical optimisation ...
- 39.6k
5
votes
What are the limitations of the hill climbing algorithm and how to overcome them?
Hill climbing is not an algorithm, but a family of "local search" algorithms. Specific algorithms which fall into the category of "hill climbing" algorithms are 2-opt, 3-opt, 2.5-opt, 4-opt, or, in ...
- 39.6k
5
votes
Accepted
When should you not use the bias in a layer?
The most usual case of bias=False is in layers before/after Batch Normalization with no activators in between. The BatchNorm layer will re-center the data anyway, ...
- 2,426
4
votes
What are the techniques for detecting and preventing overfitting?
Usually you keep track of training loss and validation loss and apply proper regularization technique (such as L1, L2, dropout, DropConnect, etc.).
The more interesting technique is to observe your ...
- 341
4
votes
When should I use simulated annealing as opposed to a genetic algorithm?
Simulated Annealing vs genetic algorithm?
Simulated annealing is a materials science analogy and involves the introduction of noise to avoid search failure due to local minima. See images below. To ...
- 7,443
4
votes
Accepted
Could error surface shape be useful to detect which local minima is better for generalization?
In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations:
what the author of the question provides us with is the Loss ...
- 799
4
votes
In deep learning, is it possible to use discontinuous activation functions?
Even the first artificial neural network - Rosenblatt's perceptron [1] had a discontinuous activation function. That network is in introductory chapters of many textbooks about AI. For example, ...
4
votes
Accepted
Why is second-order backpropagation useful?
Second-order optimization algorithms like Hessian optimization have more information on the curvature of the loss function, so converge much, much faster than first-order optimization algorithms like ...
- 1,510
4
votes
Accepted
How are these equations of SGD with momentum equivalent?
The first two equations are equivalent. The last equation can be equivalent if you scale $\alpha$ appropriately.
Equation 1
Consider the equation from the Stanford slide:
$$
v_{t}=\rho v_{t-1}+\nabla ...
- 1,510
3
votes
Accepted
Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?
Has this been done?
Difficult to prove a negative, but I suspect although plenty of research has been done into finding ideal learning rate values (the need for learning rate at all is an annoyance), ...
- 30.3k
3
votes
Accepted
Can I compute the fitness of an agent based on a low number of runs of the game?
You can probably get away with a relatively low X for two reasons:
The Central Limit Theorem. This tells us that the accuracy in the estimate of an agent's fitness will improve as the square root of ...
- 9,207
3
votes
Accepted
Maximizing or Minimizing in Trust Region Policy Optimization?
The differences you have observed between the two different versions of the TRPO paper are due to different formalizations of the problem and the objective.
In the first version of the paper you ...
- 10.1k
Only top scored, non community-wiki answers of a minimum length are eligible