When a neural network learns something from a data set, we are left with a bunch of weights which represent some approximation of knowledge about the world. Although different data sets or even different runs of the same NN might yield completely different sets of weights, the resulting equations must be mathematically similar (linear combinations, rotations, etc.). Since we usually build NNs to model a particular concrete task (identify cats, pedestrians, tumors, etc.), it seems that we are generally satisfied to let the network continue to act as a black box.

Now, I understand that there is a push for "understandability" of NNs, other ML techniques, etc. But this is not quite what I'm getting at. It seems to me that given a bunch of data points recording the behavior of charged particles, one could effectively recover Maxwell's laws using a sufficiently advanced NN. Perhaps that requires NNs which are much more sophisticated than what we have today. But it illustrates the thing I am interested in: NNs could, in my mind, be teaching us general truths about the world if we took the time to analyze and simplify the formulae that they give us1.

For instance, there must be hundreds, if not thousands of NNs which have been trained on visual recognition tasks that end up learning many of the same sub-skills, to put it a bit anthropomorphically. I recently read about gauge CNNs, but this goes the other way: we start with what we know and then bake it into the network.

Has anyone attempted to go the opposite way? Either:

  1. Take a bunch of similar NNs and analyze what they have in common to extract general formulae about the focus area2
  2. Carefully inspect the structure of a well-trained NN to directly extract the "Maxwell's equations" which might be hiding in them?

1 Imagine if we built a NN to learn Newtonian mechanics just to compute a simple ballistic trajectory. It could surely be done, but would also be a massive waste of resources. We have nice, neat equations for ballistic motion, given to us by the "original neural networks", so we use those.

2 E.g., surely the set of all visual NNs have collectively discovered near-optimal algorithms for edge/line/orientation detection, etc.). This could perhaps be done with the assistance of a meta-ML algorithm (like, clustering over NN weight matrices?).

  • 1
    $\begingroup$ Hi and welcome to this community! You say "Carefully inspect the structure of a well-trained NN to directly extract the "Maxwell's equations", but a neural network represents a function (or a set of functions). How do you plan to transform a function into an equation? Also, some neural networks have thousands or millions of parameters. How do you suggest inspecting the structure of such a big (trained) neural network? Furthermore, can you clarify how this is different than XAI? $\endgroup$
    – nbro
    Jan 9 '20 at 23:34
  • $\begingroup$ My understanding of XAI is that it's like the SQL EXPLAIN PLAN for NNs. "Justify your answer, but don't give me the full theory." Whereas, extracting the approximated function would be like discovering the query engine itself. How would you rediscover Newton's laws? You might say: "Ok, we've got some observational data on moving objects with known masses, locations and velocities. Let's make the network predict accelerations or forces or momenta." (or permute the observables) Do something analogous for other "laws". As for "how"? That's what the experts are for, right? ;) $\endgroup$ Jan 10 '20 at 0:43
  • $\begingroup$ To be more precise, XAI suggests to me that we can rationalize a particular (input, output) pair, as opposed to describing an over-arching framework which could effectively replace the NN itself as a computational shortcut (in exactly the same way as Newton's laws could replace a "ballistic NN"). As for analyzing overly large networks, we have ML tools to help us discover patterns in such things, no? If I could tell you exactly how to do it, I'd just do it myself. ;) $\endgroup$ Jan 10 '20 at 0:47
  • 1
    $\begingroup$ It's still unclear what you're looking for. You say "As for analyzing overly large networks, we have ML tools to help us discover patterns in such things, no?", so do you want to find encoded equations in the structure of a trained NN with another pattern recognition model? Or are you looking for an artificial scientist? Then, no, there is no artificial scientist (AFAIK). As I said, neural networks represent functions and not equations. The closest thing to what you're talking about that I recall is the neural programmer-interpreter (and similar stuff). $\endgroup$
    – nbro
    Jan 10 '20 at 1:13
  • $\begingroup$ Every NN approximates some function, but only represents said function implicitly. The first step would be to develop a process which discovers an explicit, algebraic/analytical form for said function, perhaps by starting with simple candidates and iteratively enriching them until they converge on the NN's representation. Obviously, the hyperplane representing the ordinary least squares regression of the NN function would be the simplest starting point, and one could add higher-order terms until they stopped improving the fit. $\endgroup$ Jan 10 '20 at 3:01