How to distinguish AI modeling from implementation?

Question

Quote from this Eric's meta post about modelling and implementation:

They are not exactly the same, although strongly related. This was a very difficult lesson to learn among mathematicians and early programmers, notably in the 70s (mathematical proofs can demand a lot of non-trivial programming work to make them "computable", as in runnable on a computer).

If they're not the same, what is the difference?

How we can say when we're talking about AI implementation, and when about modelling? It's suggested above it's not easy task. So where we can draw the line when we talk about it?

I'm asking in general, not specifically for this site, that's why I haven't posted question in meta

Answer 1

One good way of differentiating modelling and implementation is to consider that models occupy a much higher level of abstraction.

To continue with the mathematical example: even though experimental mathematics might be dependent on computation, the program can be considered as one possible realization of the necessary conditions of a more abstract existence proof.

Over the last 25 years, software engineering methodologies have become quite good at separating models and implementations, e.g. by using interfaces/typeclasses/abstract base classes to define constraints on behavior that is concretely realized by the implementation of derived classes.

AI has always been a battle between the 'neats and the scruffies'. Neats tend to prefer working 'top down' from clean abstractions, 'scruffies' like to work 'bottom up', and 'bang the bits' of the implementation together, to see what happens.

Of course, in practice, interplay between both styles is necessary, but AI as a science progresses when we abstract mechanisms away from specific implementations into their most general (and hence re-useable) form.

Answer 2

In AI (but in general too, I believe), a simplification is that modeling is more akin to Mathematics (and related hard sciences involved, like Physics and... Computer Science), and implementation to Software Engineering.

Let's take a concrete example, really outside of AI: Find the minimum value of a given polynomial, if it exists.

The Mathematician will derivate the polynomial, find the zeros, and checkout convexity to find a minimum (if there is any zero). This procedure is very standard---some will say straightforward. It relies on a body of knowledge and an abstraction level that is appropriate for manual proof.

The Software Engineer approach is actually way longer to explain, and I am going to skip it. The point is that the body of knowledge is related but different: We have to find now a step-by-step procedure for the computer to achieve the result. The Mathematician one could be implemented directly in MathLab, almost verbatim, but we assume MathLab. And to build MathLab, we are back to the problem of making a procedure the computer can execute. We could for example base a procedure on Euler's method to find roots (a "simple" approach that closes on roots step after step), etc.

Simple mathematical operations can be quite complex to implement on a computer. Perhaps the most famous is random-number generation. Mathematically, the concept is pure and clear. Generating an actual random-number is more elusive than it looks, to the point it calls for new models and new implementations...

A concrete example from history: Neural networks. In the 80s and 90s, NNs were weighted graphs that could be executed on computers using graph libraries or similar foundation libs. Choosing the weights was challenging. One day the back-propagation learning model was introduced to automated the choice of weights. The model relied on a procedure dedicated to NNs, using a terminology like partial derivates, gradient descent, chain rules, etc. And later then, clever engineers created libraries to automate the back-propagation procedure. The libraries can be somewhat far from the original model, as engineers learn how to make it computable, even faster (i.e. optimization, approximations/truncations).