Questions tagged [optimization]

For questions about implementing and improving optimization algorithms used in creating AI programs, or optimization in general.

Filter by
Sorted by
Tagged with
5
votes
3answers
901 views

What is the actual learning algorithm: back-propagation or gradient descent?

What is the actual learning algorithm: back-propagation or gradient descent (or, in general, the optimization algorithm)? I am reading through chapter 8 of Parallel Distributed Processing hand book ...
10
votes
1answer
728 views

What are the implications of the “No Free Lunch” theorem for machine learning?

The No Free Lunch (NFL) theorem states (see the paper Coevolutionary Free Lunches by David H. Wolpert and William G. Macready) any two algorithms are equivalent when their performance is averaged ...
11
votes
1answer
676 views

What are Hyper-heuristics?

I wanted to know what the differences between hyper-heuristics and meta-heuristics are, and what their main applications are. Which problems are suited to be solved by Hyper-heuristics?
4
votes
1answer
204 views

What is the difference between reinforcement learning and evolutionary algorithms?

What is the difference between reinforcement learning (RL) and evolutionary algorithms (EA)? I am trying to understand the basics of RL, but I do not yet have practical experience with RL. I know ...
8
votes
2answers
2k views

Why should the number of neurons in a hidden layer be a power of 2?

I have read somewhere on the web (I lost the reference) that the number of units (or neurons) in a hidden layer should be a power of 2 because it helps the learning algorithm to converge faster. Is ...
5
votes
1answer
3k views

How to avoid falling into the “local minima” trap?

How do I avoid my gradient descent algorithm into falling into the "local minima" trap while backpropogating on my neural network? Are there any methods which help me avoid it?
2
votes
0answers
100 views

Is a very powerful oracle sufficient to trigger the AI singularity?

Lets say we have a oracle $S$ that, given any function $F$ and desired output $y$, can find an input $x$ that causes $F$ to output $y$ if it exists, or otherwise returns nil. I.e.: $$S(F, y) = x \...
8
votes
2answers
292 views

Why does a one-layer hidden network get more robust to poor initialization with growing number of hidden neurons?

In a nutshell: I want to understand why a one hidden layer neural network converges to a good minimum more reliably when a larger number of hidden neurons is used. Below a more detailed explanation of ...
5
votes
1answer
146 views

What effect does batch norm have on the gradient?

Batch norm is a technique where they essentially standardize the activations at each layer, before passing it on to the next layer. Naturally, this will affect the gradient through the network. I have ...
5
votes
2answers
547 views

In deep learning, is it possible to use discontinuous activation functions?

In deep learning, is it possible to use discontinuous activation functions (e.g. one with jump discontinuity)? (My guess: for example, ReLU is non-differentiable at a single point, but it still has a ...
2
votes
1answer
65 views

Method to check goodness of combinatorial optimization algorithm implementation

How do I check which algorithm solves my problem best? Given a optimaization problem, I apply different well known optimization algorithms (genetic algorithm, simulated annealing, ant colony etc.) to ...
1
vote
1answer
66 views

Is an optimization algorithm equivalent to a neural network?

Is an optimization algorithm equivalent to a neural network?