Do scientists know what is happening inside artificial neural networks?

Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant? Do they understand the process behind this (e.g. what is happening inside and how it works exactly), or it is a subject of debate?

For example this study says:

However there is no clear understanding of why they perform so well, or how they might be improved.

So does this mean that scientists actually don't know how complex convolutional network models work?

• "why they perform so well" - they don't really perform SO well. As with most new technology, failures are underreported. – Tomáš Zato - Reinstate Monica May 31 '19 at 12:02
• Take a look at this question too. – nbro Nov 15 '20 at 15:01
• – kenorb Jan 22 at 20:51

There are many approaches that aim to make a trained neural network more interpretable and less like a "black box", specifically convolutional neural networks that you've mentioned.

Visualizing the activations and layer weights

Activations visualization is the first obvious and straight-forward one. For ReLU networks, the activations usually start out looking relatively blobby and dense, but as the training progresses the activations usually become more sparse (most values are zero) and localized. This sometimes shows what exactly a particular layer is focused on when it sees an image.

Another great work on activations that I'd like to mention is deepvis that shows reaction of every neuron at each layer, including pooling and normalization layers. Here's how they describe it:

In short, we’ve gathered a few different methods that allow you to “triangulate” what feature a neuron has learned, which can help you better understand how DNNs work.

The second common strategy is to visualize the weights (filters). These are usually most interpretable on the first CONV layer which is looking directly at the raw pixel data, but it is possible to also show the filter weights deeper in the network. For example, the first layer usually learns gabor-like filters that basically detect edges and blobs.

Occlusion experiments

Here's the idea. Suppose that a ConvNet classifies an image as a dog. How can we be certain that it’s actually picking up on the dog in the image as opposed to some contextual cues from the background or some other miscellaneous object?

One way of investigating which part of the image some classification prediction is coming from is by plotting the probability of the class of interest (e.g. dog class) as a function of the position of an occluder object. If we iterate over regions of the image, replace it with all zeros and check the classification result, we can build a 2-dimensional heat map of what's most important for the network on a particular image. This approach has been used in Matthew Zeiler’s Visualizing and Understanding Convolutional Networks (that you refer to in your question):

Deconvolution

Another approach is to synthesize an image that causes a particular neuron to fire, basically what the neuron is looking for. The idea is to compute the gradient with respect to the image, instead of the usual gradient with respect to the weights. So you pick a layer, set the gradient there to be all zero except for one for one neuron and backprop to the image.

Deconv actually does something called guided backpropagation to make a nicer looking image, but it's just a detail.

Similar approaches to other neural networks

Highly recommend this post by Andrej Karpathy, in which he plays a lot with Recurrent Neural Networks (RNN). In the end, he applies a similar technique to see what the neurons actually learn:

The neuron highlighted in this image seems to get very excited about URLs and turns off outside of the URLs. The LSTM is likely using this neuron to remember if it is inside a URL or not.

Conclusion

I've mentioned only a small fraction of results in this area of research. It's pretty active and new methods that shed light to the neural network inner workings appear each year.

To answer your question, there's always something that scientists don't know yet, but in many cases they have a good picture (literary) of what's going on inside and can answer many particular questions.

To me the quote from your question simply highlights the importance of research of not only accuracy improvement, but the inner structure of the network as well. As Matt Zieler tells in this talk, sometimes a good visualization can lead, in turn, to better accuracy.

It depends on what you mean by "know what is happening".

Conceptually, yes: ANN perform nonlinear regression. The actual expression represented by the weight matrix/activation function(s) of an ANN can be explicitly expanded in symbolic form (e.g. containing sub-expressions such as $$1/1+e^{1/1+e^{\dots}}$$).

However, if by 'know' you mean predicting the output of some specific (black box) ANN, by some other means, then the obstacle is the presence of chaos in a ANN that has high degrees of freedom.

Here's also some relatively recent work by Hod Lipson on understanding ANNs through visualisation.

Model interpretability is a hyper-active and hyper-hot area of current research (think of holy grail, or something), which has been brought forward lately not least due to the (often tremendous) success of deep learning models in various tasks; these models are currently only black boxes, and we naturally feel uncomfortable about it...

Here are some general (and recent, as of Dec 2017) resources on the subject:

And on a more practical level (code etc):

Lately, there has been a surge of interest to start building a more theoretical basis for deep learning neural nets. In this context, renowned statistician and compressive sensing pioneer David Donoho has very recently (fall 2017) started offering a course at Stanford, Theories of Deep Learning (STATS 385), with almost all the material available online; it is highly recommended...

NOTE: I do no longer keep this answer updated; for updates, see my answer in Which explainable artificial intelligence techniques are there?

• Hi. This seems to be a good answer, but you need to clean it up and organise a little bit. The first resources should be the most helpful and general. Then you can list more specific resources and research papers, IMHO. And later you can list e.g. Twitter threads or whatever. – nbro Jun 28 '19 at 14:08

I'm afraid I don't have the specific citations handy, but I have seen/heard quotes by experts like Andrew Ng and Geoffrey Hinton where they clearly say that we do not really understand neural networks. That is, we understand something of the how they work (for example, the math behind back propagation) but we don't really understand why they work. It's sort of a subtle distinction, but the point is that no, we don't understand the very deepest details of how exactly you go from a bunch of weights, to, say, recognizing a cat playing with a ball.

At least in terms of image recognition, the best explanation I've heard is that successive layers of a neural network learn more sophisticated features, composed of the more granular features from earlier levels. That is to say, the first layer might recognize "edges" or "straight lines". The next layer might then learn geometric shapes like "box", or "triangle", and then a higher layer might learn "nose" or "eye" based on those earlier features, and then a higher level layer still learns "face" made up from "eye", "nose", "jaw", etc. But even that, as I understand it, is still hypothetical and/or not understood in complete detail.

• I'd be interested to read the actual quotes. At the broadest conceptual level, the why is "They are Universal function approximators trained to reduce the error in a regression problem". – NietzscheanAI Aug 10 '16 at 16:09
• I'll see if I can track them down. I'm pretty sure the quote from Geoffrey Hinton that I'm thinking of is in a video.. either from his Coursera Class or some video he has up on Youtube. If I can find it, I'll edit my answer and link it in. – mindcrime Aug 10 '16 at 17:31
• I haven't forgotten. I'll try to find them when I have a little free time. I think at least one of the ones I'm thinking of was from a video that's part of a Coursera course. – mindcrime Aug 16 '16 at 16:28
• This study can help to put same references: 'However there is no clear understanding of why they perform so well, or how they might be improved'. – kenorb Aug 17 '16 at 17:34

Not sure if this is what you are searching for, but google extracted images from networks when they were fed with white noise.

This kind of represents what the network knows.

Here is an answer by Carlos E. Perez to the question What is theory behind deep learning?

[...]

The underlying mathematics of Deep Learning has been in existence for several decades, however the impressive results that we see today are part a consequence of much faster hardware, more data and incremental improvements in methods.

Deep Learning in general can be framed as optimization problem where the objective is a function of the model error. This optimization problem is very difficult to solve consider that the parameter space of the model (i.e. weights of the neural network) leads to a problem in extremely high dimension. An optimization algorithm could take a very long time to explore this space. Furthermore, there was an unverified belief that the problem was non-convex and computation would forever be stuck in local minima.

[...]

The theory of why machines actually converge to an attractor or in other words learn to recognize complex patterns is still unknown.

To sum up: we have some ideas, but we're not quite sure.

Do scientists know what is happening inside artificial neural networks?

YES

Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant?

I guess "to know from the kitchen" means "to know in detail"?

Let me give you a series of analogies:

1. Does an airplane engineer know from the kitchen what happens inside the airplane?
2. Does a chip designer know in detail what happens in the chip (s)he designed?
3. Does a civil engineer know everything about the house he constructed?

The devil is in the detail, but a crucial point here is that it's about artificial structures. They don't randomly appear. You need a lot of knowledge to get anything useful. For Neural Networks, I would say it took roughly 40 years from the publication of the key idea (Rosenblatt perceptron, 1957) to the first application (US Postal Service, 1989). And from there again 13 years of active reserach to really impressive systems (ImageNet 2012).

What we know super well is how the training works. Because it needs to be implemented. So on a very small structure, we know it in detail.

Think of computers. The chip designers know very well how their chip works. But they will likely only have a very rough idea how the Linux operating system works.

Another example is physics and chemistry: Physics describes the core forces of the universe. Does that mean they know everything about chemistry as well? Hell no! A "perfect" physicist can explain everything in chemistry ... but it would be pretty much useless. He would need a lot more information, not be able to skip the irrelevant parts. Simply because he "zoomed in" too much - considers details which are in practice neither interesting nor important. Please note that the knowledge of the physicist is not wrong. Maybe one could even deduce the knowledge from the chemist from it. But this "high-level" understanding of molecule interaction is missing.

The key insight from those two examples are abstraction layers: You can build complexity from simple structures.

What else?

We know well what is in principle achievable with the neural networks we design:

• A neural network designed to play Go - no matter how sophisticated - will never even be able to play chess. You can, of course, add another abstraction layer around it and combine things. But this approach needs humans.
• A neural network designed for distinguishing dogs from cats which has only seen pudels and Persian cats will likely perform really bad when it has to decide for Yorkshire Terriers.

Oh, and of course we have analytical approaches for neural networks. I wrote my masters thesis about Analysis and Optimization of Convolutional Neural Network Architectures. In this context LIME (Local Interpretable Model-Agnostic Explanations) is nice:

• Most of them are influenced by biological models..So saying Scientists built NN's as a function of a problem is kind of hard to believe... Especially when no one has any idea why a particular architecture or a particular set of hyperparameters work well for a given problem...I am not talking about the exact hyperparameters but none seem to have a general sense of what approximate hyperparameters might work for a given problem (problem is well defined)..So no scientists don't know what's happening inside a NN. – user9947 Sep 25 '18 at 17:32
• Think of early day automobile / aircraft engineers. Would you say they don't know what's happening inside their aircraft / automobile, because they didn't built them, because their shape was not aerodynamic? – Martin Thoma Sep 25 '18 at 18:07
• Ofc...Not knowing something due to lack of technology...Is something different than not knowing theoretically..I believe it was technology in airplanes case..While here we are not able to process mathematically..So forget technology – user9947 Sep 25 '18 at 18:26