Do scientists know what is happening inside artificial neural networks?

Question

Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant? Do they understand the process behind this (e.g. what is happening inside and how it works exactly), or it is a subject of debate?

For example this study says:

However there is no clear understanding of why they perform so well, or how they might be improved.

So does this mean that scientists actually don't know how complex convolutional network models work?

Answer 1

There are many approaches that aim to make a trained neural network more interpretable and less like a "black box", specifically convolutional neural networks that you've mentioned.

Visualizing the activations and layer weights

Activations visualization is the first obvious and straight-forward one. For ReLU networks, the activations usually start out looking relatively blobby and dense, but as the training progresses the activations usually become more sparse (most values are zero) and localized. This sometimes shows what exactly a particular layer is focused on when it sees an image.

Another great work on activations that I'd like to mention is deepvis that shows reaction of every neuron at each layer, including pooling and normalization layers. Here's how they describe it:

In short, we’ve gathered a few different methods that allow you to “triangulate” what feature a neuron has learned, which can help you better understand how DNNs work.

The second common strategy is to visualize the weights (filters). These are usually most interpretable on the first CONV layer which is looking directly at the raw pixel data, but it is possible to also show the filter weights deeper in the network. For example, the first layer usually learns gabor-like filters that basically detect edges and blobs.

Occlusion experiments

Here's the idea. Suppose that a ConvNet classifies an image as a dog. How can we be certain that it’s actually picking up on the dog in the image as opposed to some contextual cues from the background or some other miscellaneous object?

One way of investigating which part of the image some classification prediction is coming from is by plotting the probability of the class of interest (e.g. dog class) as a function of the position of an occluder object. If we iterate over regions of the image, replace it with all zeros and check the classification result, we can build a 2-dimensional heat map of what's most important for the network on a particular image. This approach has been used in Matthew Zeiler’s Visualizing and Understanding Convolutional Networks (that you refer to in your question):

Deconvolution

Another approach is to synthesize an image that causes a particular neuron to fire, basically what the neuron is looking for. The idea is to compute the gradient with respect to the image, instead of the usual gradient with respect to the weights. So you pick a layer, set the gradient there to be all zero except for one for one neuron and backprop to the image.

Deconv actually does something called guided backpropagation to make a nicer looking image, but it's just a detail.

Similar approaches to other neural networks

Highly recommend this post by Andrej Karpathy, in which he plays a lot with Recurrent Neural Networks (RNN). In the end, he applies a similar technique to see what the neurons actually learn:

The neuron highlighted in this image seems to get very excited about URLs and turns off outside of the URLs. The LSTM is likely using this neuron to remember if it is inside a URL or not.

Conclusion

I've mentioned only a small fraction of results in this area of research. It's pretty active and new methods that shed light to the neural network inner workings appear each year.

To answer your question, there's always something that scientists don't know yet, but in many cases they have a good picture (literary) of what's going on inside and can answer many particular questions.

To me the quote from your question simply highlights the importance of research of not only accuracy improvement, but the inner structure of the network as well. As Matt Zieler tells in this talk, sometimes a good visualization can lead, in turn, to better accuracy.

Answer 2

It depends on what you mean by "know what is happening".

Conceptually, yes: ANN perform nonlinear regression. The actual expression represented by the weight matrix/activation function(s) of an ANN can be explicitly expanded in symbolic form (e.g. containing sub-expressions such as $1/1+e^{1/1+e^{\dots}}$).

However, if by 'know' you mean predicting the output of some specific (black box) ANN, by some other means, then the obstacle is the presence of chaos in a ANN that has high degrees of freedom.

Here's also some relatively recent work by Hod Lipson on understanding ANNs through visualisation.

Answer 3

Short answer is no.

Model interpretability is a hyper-active and hyper-hot area of current research (think of holy grail, or something), which has been brought forward lately not least due to the (often tremendous) success of deep learning models in various tasks; these models are currently only black boxes, and we naturally feel uncomfortable about it...

Here are some general (and recent, as of Dec 2017) resources on the subject:

• A recent (July 2017) article in Science provides a nice overview of the current status & research: How AI detectives are cracking open the black box of deep learning (no in-text links, but googling names & terms will pay off)

• DARPA itself is currently running a program on Explainable Artificial Intelligence (XAI)

• There was a workshop in NIPS 2016 on Interpretable Machine Learning for Complex Systems, as well as an ICML 2017 tutorial on Interpretable Machine Learning by Been Kim of Google Brain.

And on a more practical level (code etc):

• The What-If tool by Google, a brand new (September 2018) feature of the open-source TensorBoard web application, which let users analyze an ML model without writing code (project page, blog post)

• The Layer-wise Relevance Propagation (LRP) toolbox for neural networks (paper, project page, code, TF Slim wrapper)

• FairML: Auditing Black-Box Predictive Models, by Cloudera Fast Forward Labs (blog post, paper, code)

• LIME: Local Interpretable Model-agnostic Explanations (paper, code, blog post, R port)

• A very recent (November 2017) paper by Geoff Hinton, Distilling a Neural Network Into a Soft Decision Tree, with an independent PyTorch implementation

• SHAP: A Unified Approach to Interpreting Model Predictions (paper, authors' Python code, R package)

• Interpretable Convolutional Neural Networks (paper, authors' code)

• Lucid, a collection of infrastructure and tools for research in neural network interpretability by Google (code; papers: Feature Visualization, The Building Blocks of Interpretability)

• Transparecy-by-Design (TbD) networks (paper, code, demo)

• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability (paper, code, Google blog post)

• TCAV: Testing with Concept Activation Vectors (ICML 2018 paper, Tensorflow code)

• Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization (paper, authors' Torch code, Tensorflow code, PyTorch code, Keras example notebook)

• Network Dissection: Quantifying Interpretability of Deep Visual Representations, by MIT CSAIL (project page, Caffe code, PyTorch port)

• GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, by MIT CSAIL (project page, with links to paper & code)

• Explain to Fix: A Framework to Interpret and Correct DNN Object Detector Predictions (paper, code)

• InterpretML by Microsoft (code still in alpha)

• Anchors: High-Precision Model-Agnostic Explanations (paper, code)

• Diverse Counterfactual Explanations (DiCE) by Microsoft (paper, code, blog post)

• Axiom-based Grad-CAM (XGrad-CAM): Towards Accurate Visualization and Explanation of CNNs, a refinement of the existing Grad-CAM method (paper, code)

Lately, there has been a surge of interest to start building a more theoretical basis for deep learning neural nets. In this context, renowned statistician and compressive sensing pioneer David Donoho has very recently (fall 2017) started offering a course at Stanford, Theories of Deep Learning (STATS 385), with almost all the material available online; it is highly recommended...

UPDATES:

• Interpretable Machine Learning, an online Gitbook by Christoph Molnar with R code (although in mostly covers algorithms other than neural networks)
• A Twitter thread, linking to several interpretation tools available for R.
• A short (4 hrs) online course by Kaggle, Machine Learning Explainability, and the accompanying blog post
• A new ICML 2018 Tutorial, Toward theoretical understanding of deep learning, by Sanjeev Arora
• A whole bunch of resources in the Awesome Machine Learning Interpretability repo

NOTE: I do no longer keep this answer updated; for updates, see my answer in Which explainable artificial intelligence techniques are there?

Answer 4

I'm afraid I don't have the specific citations handy, but I have seen/heard quotes by experts like Andrew Ng and Geoffrey Hinton where they clearly say that we do not really understand neural networks. That is, we understand something of the how they work (for example, the math behind back propagation) but we don't really understand why they work. It's sort of a subtle distinction, but the point is that no, we don't understand the very deepest details of how exactly you go from a bunch of weights, to, say, recognizing a cat playing with a ball.

At least in terms of image recognition, the best explanation I've heard is that successive layers of a neural network learn more sophisticated features, composed of the more granular features from earlier levels. That is to say, the first layer might recognize "edges" or "straight lines". The next layer might then learn geometric shapes like "box", or "triangle", and then a higher layer might learn "nose" or "eye" based on those earlier features, and then a higher level layer still learns "face" made up from "eye", "nose", "jaw", etc. But even that, as I understand it, is still hypothetical and/or not understood in complete detail.

Answer 5

Not sure if this is what you are searching for, but google extracted images from networks when they were fed with white noise.

See Inceptionism: Going Deeper into Neural Networks (Google Research Blog).

This kind of represents what the network knows.

Answer 6

Here is an answer by Carlos E. Perez to the question What is theory behind deep learning?

[...]

The underlying mathematics of Deep Learning has been in existence for several decades, however the impressive results that we see today are part a consequence of much faster hardware, more data and incremental improvements in methods.

Deep Learning in general can be framed as optimization problem where the objective is a function of the model error. This optimization problem is very difficult to solve consider that the parameter space of the model (i.e. weights of the neural network) leads to a problem in extremely high dimension. An optimization algorithm could take a very long time to explore this space. Furthermore, there was an unverified belief that the problem was non-convex and computation would forever be stuck in local minima.

[...]

The theory of why machines actually converge to an attractor or in other words learn to recognize complex patterns is still unknown.

To sum up: we have some ideas, but we're not quite sure.

Answer 7

Do scientists know what is happening inside artificial neural networks?

YES

Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant?

I guess "to know from the kitchen" means "to know in detail"?

Let me give you a series of analogies:

1. Does an airplane engineer know from the kitchen what happens inside the airplane?
2. Does a chip designer know in detail what happens in the chip (s)he designed?
3. Does a civil engineer know everything about the house he constructed?

The devil is in the detail, but a crucial point here is that it's about artificial structures. They don't randomly appear. You need a lot of knowledge to get anything useful. For Neural Networks, I would say it took roughly 40 years from the publication of the key idea (Rosenblatt perceptron, 1957) to the first application (US Postal Service, 1989). And from there again 13 years of active reserach to really impressive systems (ImageNet 2012).

What we know super well is how the training works. Because it needs to be implemented. So on a very small structure, we know it in detail.

Think of computers. The chip designers know very well how their chip works. But they will likely only have a very rough idea how the Linux operating system works.

Another example is physics and chemistry: Physics describes the core forces of the universe. Does that mean they know everything about chemistry as well? Hell no! A "perfect" physicist can explain everything in chemistry ... but it would be pretty much useless. He would need a lot more information, not be able to skip the irrelevant parts. Simply because he "zoomed in" too much - considers details which are in practice neither interesting nor important. Please note that the knowledge of the physicist is not wrong. Maybe one could even deduce the knowledge from the chemist from it. But this "high-level" understanding of molecule interaction is missing.

The key insight from those two examples are abstraction layers: You can build complexity from simple structures.

What else?

We know well what is in principle achievable with the neural networks we design:

• A neural network designed to play Go - no matter how sophisticated - will never even be able to play chess. You can, of course, add another abstraction layer around it and combine things. But this approach needs humans.
• A neural network designed for distinguishing dogs from cats which has only seen pudels and Persian cats will likely perform really bad when it has to decide for Yorkshire Terriers.

Oh, and of course we have analytical approaches for neural networks. I wrote my masters thesis about Analysis and Optimization of Convolutional Neural Network Architectures. In this context LIME (Local Interpretable Model-Agnostic Explanations) is nice: