14 votes
Accepted

Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. I ...
user avatar
12 votes
Accepted

What are the implications of the "No Free Lunch" theorem for machine learning?

This is a really common reaction after first encountering the No Free Lunch theorems (NFLs). The one for machine learning is especially unintuitive, because it flies in the face of everything that's ...
user avatar
10 votes

Can artificial intelligence be thought of as optimization?

A good answer to this question depends on what you want to use the labels for. When I think about "optimization," I think about a solution space and a cost function; that is, there are many possible ...
user avatar
10 votes
Accepted

Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch

I see no reason why decaying learning rates should create the kinds of jumps in losses that you are observing. It should "slow down" how quickly you "move", which in the case of a loss that otherwise ...
user avatar
  • 9,379
8 votes
Accepted

What are hyper-heuristics, and how are they different from meta-heuristics?

TL:DR: Hyper-heuristics are metaheuristics, suited for solving the same kind of optimization problems, but (in principle) affording a "rapid prototyping" approach for non-expert practitioners. In ...
user avatar
8 votes
Accepted

How to avoid falling into the "local minima" trap?

There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include: Probabalistically accepting worse solutions in the hope that this will ...
user avatar
8 votes
Accepted

What is the difference between reinforcement learning and evolutionary algorithms?

Evolutionary algorithms (EAs) are a family of algorithms inspired by the biological evolution that can be used to solve (constrained or not) optimization problems where the function that needs to be ...
user avatar
  • 33.8k
8 votes
Accepted

Why is the learning rate generally beneath 1?

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition $$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$ where $a_t$ is the learning rate at iteration $t$, ...
user avatar
  • 33.8k
7 votes

What is the actual learning algorithm: back-propagation or gradient descent?

You can run gradient descent without back propagation, in some cases: Simple structures such as linear or logistic regression, where the gradients can be calculated directly from the inputs and cost ...
user avatar
7 votes
Accepted

Why is the perceptron criterion function differentiable?

$\max(-y_i(w x_i), 0)$ is not partial derivable respect $w$ if $w x_i=0$. Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval ...
user avatar
6 votes
Accepted

If Deep Learning is non convex, then why do use a convex loss function?

Well, you are definitely mixing two different things. Here are those bits: The function that deep learning approximates is basically a function that best fits the INPUT DATA points. You should not ...
user avatar
6 votes
Accepted

What are the limitations of the hill climbing algorithm and how to overcome them?

As @nbro has already said that Hill Climbing is a family of local search algorithms. So, when you said Hill Climbing in the question I have assumed you are talking about the standard hill climbing. ...
user avatar
  • 1,953
6 votes
Accepted

How is it possible that the MSE used to train neural networks with gradient descent has multiple local minima?

$g(x) = x^2$ is indeed a parabola and thus has just one optimum. However, the $\text{MSE}(\boldsymbol{x}, \boldsymbol{y}) = \sum_i (y_i - f(x_i))^2$, where $\boldsymbol{x}$ are the inputs, $\...
user avatar
  • 33.8k
5 votes

Can artificial intelligence be thought of as optimization?

I can offer two (at first sight, conflicting) perspectives on this: Firstly: If the letter string 'abc' becomes 'abd' what would "doing the same thing" to 'ijk' look like? This is just one example ...
user avatar
5 votes

What is the actual learning algorithm: back-propagation or gradient descent?

Gradient descent (GD) is an optimisation algorithm, that is, it is used to find a (local) minimum of a multi-variable and differentiable function $f$. GD is an iterative and numerical optimisation ...
user avatar
  • 33.8k
5 votes

What are the limitations of the hill climbing algorithm and how to overcome them?

Hill climbing is not an algorithm, but a family of "local search" algorithms. Specific algorithms which fall into the category of "hill climbing" algorithms are 2-opt, 3-opt, 2.5-opt, 4-opt, or, in ...
user avatar
  • 33.8k
5 votes
Accepted

When should we use algorithms like Adam as opposed to SGD?

Empirically, I observed that algorithms like Adam and RMSProp tended to give me a final higher performance (in my case, the accuracy) on (the validation dataset) with respect to SGD. However, I also ...
user avatar
  • 33.8k
4 votes

What are the techniques for detecting and preventing overfitting?

Usually you keep track of training loss and validation loss and apply proper regularization technique (such as L1, L2, dropout, DropConnect, etc.). The more interesting technique is to observe your ...
user avatar
4 votes
Accepted

What AI technique should I use to assign a person to a task?

What you have could be well described as a Task Allocation problem, which is studied as part of the planning subfield of AI. Chapters 10 & 11 of Russell & Norvig provide a good overview of ...
user avatar
4 votes
Accepted

What is an objective function?

The "objective function" is the function that you want to minimise or maximise in your problem. The expression "objective function" is used in several different contexts (e.g. machine learning or ...
user avatar
  • 33.8k
4 votes
Accepted

Could error surface shape be useful to detect which local minima is better for generalization?

In general I agree with @nbro answer, nevertheless sticking strictly to this specific question I'd like to share some speculations: what the author of the question provides us with is the Loss ...
user avatar
4 votes

In deep learning, is it possible to use discontinuous activation functions?

Even the first artificial neural network - Rosenblatt's perceptron [1] had a discontinuous activation function. That network is in introductory chapters of many textbooks about AI. For example, ...
user avatar
4 votes
Accepted

How are these equations of SGD with momentum equivalent?

The first two equations are equivalent. The last equation can be equivalent if you scale $\alpha$ appropriately. Equation 1 Consider the equation from the Stanford slide: $$ v_{t}=\rho v_{t-1}+\nabla ...
user avatar
  • 1,230
4 votes

Why is gradient descent used over the conjugate gradient method?

When dealing with optimization problems, a fundamental distinction is whether the objective is a (deterministic) function, or an expectation of some function. I will refer to these cases as the ...
user avatar
  • 876
3 votes
Accepted

Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?

Has this been done? Difficult to prove a negative, but I suspect although plenty of research has been done into finding ideal learning rate values (the need for learning rate at all is an annoyance), ...
user avatar
3 votes
Accepted

Can I compute the fitness of an agent based on a low number of runs of the game?

You can probably get away with a relatively low X for two reasons: The Central Limit Theorem. This tells us that the accuracy in the estimate of an agent's fitness will improve as the square root of ...
user avatar
3 votes
Accepted

Maximizing or Minimizing in Trust Region Policy Optimization?

The differences you have observed between the two different versions of the TRPO paper are due to different formalizations of the problem and the objective. In the first version of the paper you ...
user avatar
  • 9,379
3 votes
Accepted

Is there a way to define the boundaries of the optimal size of a training set?

For a finite value to be 'optimal,' typically you need some benefit from more paired up with some cost for more, and eventually the lines cross because the benefit decreases and the cost increases. ...
user avatar
3 votes

Method to check goodness of combinatorial optimization algorithm implementation

This is a very large question that could be answered in a variety of ways depending on the context. For some optimization problems operating under specific conditions you can make theoretical ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible