How can I systematically learn about the theory of neural networks?

Question

I have seen a few articles about neural nets. Mostly they went along these lines: we tried these architectures, these meta parameters, we trained it for $x$ hours on $y$ CPUs, and it gave us these results that are 0.1% better than state of the art.

What I am interested in is whether there exists (at least as a work in progress) a framework that gives some explanation why is some architecture better than other, what makes one activation function more suitable for image recognition than another, etc.

Do you have some tips about where to start? I would prefer something more systematic than a Google search (a book, a list of key articles is ideal).

Answer 1

Good question, there is a lot of work in that field. The first part before saying which machine learning algorithm is better and why is defining the problem. Is the problem an optimisation, classification, anomaly detection problem because then you need to use the appropriate machine learning algorithms. Let's assume its an optimisation problem.

Is this problem, dynamic, or static. Is this time series data? So we need to understand the problem.

Each optimisation problem has a landscape or a fitness landscape. Computer science has some nice toy landscapes.

There is a lot of work in determining the nature of the problem landscape. Have a look at K Malan's work.

Once you can identify the problem space and understand it's characteristics then you can start to identify what machine learning functions work well on what kind of landscapes. This is a totally different field of research.

For example some researchers are working on how different evolutionary algorithms handle different landscapes, or neural networks handle different landscapes.

Start by exploring the types of machine learning problems. Understand the complexity of the problem, followed by classification of machine learning algorithms for specific problem spaces.