How to decide whether a problem needs to be solved algorithmically or with machine learning techniques?

Question

There are problems (e.g. this one or this other one) that could potentially be solved easily using traditional algorithmic techniques. I think that training a neural network (or any other machine learning model) for such sorts of problems will be more time consuming, resource-intensive, and pointless.

If I want to solve a problem, how to decide whether it is better to solve algorithmically or by using NN/ML techniques? What are the pros and cons? How can this be done in a systematic way? And if I have to answer someone why I chose a particular domain, how should I answer?

Example problems are appreciated.

Answer 1

There are two different problems described in the linked question and your question: optimization and learning.

Optimization

If you are asking about optimization (the second linked question: Search minimum value with learning machine algorithm) you can have 3 different approaches:

• analytical approach
• numerial methods
• metaheuristics

As you suggest, it is usually better to try them from the first to the last one. It is common that the first approach is unfeasible for optimizing for target function, but very often you can use either mathematical optimization for some specific classes of problems (e.g. linear/quadratic programming) or iterative methods (e.g. conjugate gradient method). Only after considering this approaches it makes sense for the third class of approaches, genetic algorithms being a notable example, which is often classified as an AI approach.

Learning

If you are asking about learning, then the first linked question (Ideas on how to make a neural net learn how to split sequence into sub sequences) seems to be intended as an example. However it doesn't make clear what the problem is, as the target function seems to be obvious, so no learning is needed.

In this case it also makes sense to first try to pin down the problem mathematically and resort to machine learning if it is impossible and if you have the data (input/output examples).

Answer 2

When we apply supervised learning to a problem, we are already systematizing the approach. A human has decided that a function exists (mapping from inputs to unique output) and that the offered features are the only ones that need be considered. The learning then goes ahead to find the best solution given those constraints. Unsupervised learning is a bit more general, searching for associations or relations that might not necessarily be functions. A neural net is not yet capable of generalizing and asking for more information, it can only become more specific unless a human intervenes.

Everything depends on the detail of the problem. If it is clear that a function must exist then we can set a NN to find that function. Many other problems are more difficult - a company is losing money and you have data but halfway there was a change in CEO, so human reasoning has to be mixed in to deal with the situation. The human can modify the architecture of the NN to introduce dummy variables, but the NN cannot do this by itself.

So your answer really is "I chose this method because of the (lack of) need for me to artificially constrain the approach to the problem."

Answer 3

fwiw, with the basic, non-trivial M-game, I have no doubt that ALphaZero could tear through any human player alive in very short order. I hope that people will start experimenting with that, especially on m^n(m^n) where m > 3 and n > 2 to see how they hold up. Problem is, once you expand past n > 3 it gets very difficult for humans to play. This leads to a condition where performance of an NN on higher order M can only realistically be evaluated against other algorithms. In this context, it seems worthwhile to develop a general, classical algorithm that can evaluate any order M, regardless of efficacy of tree search in relation to the problem size, with the understanding that decision making is never presumed optimal until the gametree becomes tractable. This carries an an assumption of the same general strength across all M for the classical algorithm, because the expansion of m or n do alter the core heuristics.

From the practical standpoint, as a product designed for mobile with no assumption of connectivity, it doesn't make sense to start integrating NNs until lowest-common-denominator mobile devices have sufficient resources. The issue of package size is also important in this context--the classical algorithms require a trivial amount of code and volume. Most importantly, using classical algorithms formed of sets of heuristics and parameters allows recombination of functions to produce myriad automata of varying degrees of strength. (This can be easily accomplished by altering the size of tree search algorithms, but may only be relevant in determining which heuristics perform better under tree search restrictions.)

Finally, because M-games provide an array precise metrics, it may be worthwhile to develop core heuristic function based on human reasoning.

Answer 4

Note that "algorithmically" can refer to anything that uses an algorithm. Currently, ML systems are trained with algorithms and neural networks can be seen as algorithms (although black-box ones), so ML is also algorithmic. Everything that runs on a computer (a concrete version of a Turing machine) can be seen as an algorithm (or program)! In fact, computers were invented exactly for this purpose: to perform some algorithmic operation (i.e. a set of instructions, like a recipe).

So, by algorithmic, I assume you're referring to techniques that are typically taught in an "Algorithms and Data Structures" course for a computer science student, such as the binary search (one of the most simple and yet beautiful and useful algorithms!), which is an algorithm that, given some constraints (a sorted array), gives you an exact correct solution in $\mathcal{O}(\log n)$ time. However, I think that you are also referring to every program that is primarily based on if-then statements and loops (e.g. desktop applications, websites, etc.)

To answer your question, you first need to understand the scope of the machine learning field.

Machine learning (like statistics) is a set of techniques that attempt to learn from data. So, every problem where data is available (and you can get insight from) can potentially be solved with a machine learning technique. ML techniques typically produce approximative solutions and are typically used to solve problems where an exact solution is infeasible. However, note that machine learning isn't the only approach to solve hard problems (e.g. you can also use meta-heuristics, e.g. ant colony optimization algorithms).

If you have an algorithm that produces an exact solution (without requiring data) in polynomial time (preferably, in $\mathcal{O}(n^2)$ time), then machine learning (or any other technique that produces approximative solutions, e.g. heuristics) is quite useless.