# Tag Info

Accepted

### Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. I ...
• 26.5k
Accepted

### What are the implications of the "No Free Lunch" theorem for machine learning?

This is a really common reaction after first encountering the No Free Lunch theorems (NFLs). The one for machine learning is especially unintuitive, because it flies in the face of everything that's ...
• 9,037
Accepted

### Loss jumps abruptly when I decay the learning rate with Adam optimizer in PyTorch

I see no reason why decaying learning rates should create the kinds of jumps in losses that you are observing. It should "slow down" how quickly you "move", which in the case of a loss that otherwise ...
• 9,794
Accepted

### Why is the learning rate generally beneath 1?

If the learning rate is greater than or equal to $1$ the Robbins-Monro condition $$\sum _{{t=0}}^{{\infty }}a_{t}^{2}<\infty\label{1}\tag{1},$$ where $a_t$ is the learning rate at iteration $t$, ...
• 37k

### Can artificial intelligence be thought of as optimization?

A good answer to this question depends on what you want to use the labels for. When I think about "optimization," I think about a solution space and a cost function; that is, there are many possible ...
• 4,222
Accepted

### What are hyper-heuristics, and how are they different from meta-heuristics?

TL:DR: Hyper-heuristics are metaheuristics, suited for solving the same kind of optimization problems, but (in principle) affording a "rapid prototyping" approach for non-expert practitioners. In ...
• 7,176
Accepted

### How to avoid falling into the "local minima" trap?

There are several elementary techniques to try and move a search out of the basin of attraction of local optima. They include: Probabalistically accepting worse solutions in the hope that this will ...
• 7,176
Accepted

• 1,400
Accepted

### When should you not use the bias in a layer?

The most usual case of bias=False is in layers before/after Batch Normalization with no activators in between. The BatchNorm layer will re-center the data anyway, ...
• 2,248
Accepted

### Is a calculus or ML approach to varying learning rate as a function of loss and epoch been investigated?

Has this been done? Difficult to prove a negative, but I suspect although plenty of research has been done into finding ideal learning rate values (the need for learning rate at all is an annoyance), ...
• 26.5k
Accepted

### Can I compute the fitness of an agent based on a low number of runs of the game?

You can probably get away with a relatively low X for two reasons: The Central Limit Theorem. This tells us that the accuracy in the estimate of an agent's fitness will improve as the square root of ...
• 9,037