Questions tagged [optimization]

For questions about implementing and improving optimization algorithms used in creating AI programs, or optimization in general.

Filter by
Sorted by
Tagged with
8
votes
1answer
375 views

What are the implications of the “No Free Lunch” theorem for machine learning?

The No Free Lunch (NFL) theorem states (see the paper Coevolutionary Free Lunches by David H. Wolpert and William G. Macready) any two algorithms are equivalent when their performance is averaged ...
9
votes
1answer
569 views

What are Hyper-heuristics?

I wanted to know what the differences between hyper-heuristics and meta-heuristics are, and what their main applications are. Which problems are suited to be solved by Hyper-heuristics?
4
votes
4answers
422 views

What is the actual learning algorithm: back-propagation or gradient descent?

What is the actual learning algorithm: back-propagation or gradient descent (or, in general, the optimization algorithm)? I am reading through chapter 8 of Parallel Distributed Processing hand book ...
5
votes
1answer
3k views

How to avoid falling into the “local minima” trap?

How do I avoid my gradient descent algorithm into falling into the "local minima" trap while backpropogating on my neural network? Are there any methods which help me avoid it?
2
votes
0answers
78 views

Is a very powerful oracle sufficient to trigger the AI singularity?

Lets say we have a oracle $S$ that, given any function $F$ and desired output $y$, can find an input $x$ that causes $F$ to output $y$ if it exists, or otherwise returns nil. I.e.: $$S(F, y) = x \...
7
votes
1answer
221 views

Why does a one-layer hidden network get more robust to poor initialization with growing number of hidden neurons?

In a nutshell: I want to understand why a one hidden layer neural network converges to a good minimum more reliably when a larger number of hidden neurons is used. Below a more detailed explanation of ...
4
votes
2answers
92 views

Can we optimize an optimization algorithm?

In this answer to the question Is an optimization algorithm equivalent to a neural network?, the author stated that, in theory, there is some recurrent neural network that implements a given ...
2
votes
1answer
52 views

Method to check goodness of combinatorial optimization algorithm implementation

How do I check which algorithm solves my problem best? Given a optimaization problem, I apply different well known optimization algorithms (genetic algorithm, simulated annealing, ant colony etc.) to ...
1
vote
1answer
55 views

Is an optimization algorithm equivalent to a neural network?

Is an optimization algorithm equivalent to a neural network?