0
$\begingroup$

If a certain task T is solved by a non-learning-based method A (let's say, an optimization-based approach). We now train a machine learning model B (let's say a neural network) on the same task.

What are some metrics that we can use to compare their efficiency in terms of finding the solution (Assuming the quality of both solutions is comparable)?

$\endgroup$

1 Answer 1

1
$\begingroup$

The most generic answer to this question is:

the same metrics you use to evaluate the quality of your model during training or in test phase. (Plus the timing of inference if you're referring to computational efficiency)

And I'm not referring to any specific metric yet cause that's really task dependent. But in general if you have a model that perform a task and another algorithm that perform the same task, then you should be able to apply both to the same set of data, compute whatever metric is suitable to evaluate the performance on the task, and compare the two scores. Let me stress out that the test instances should be the same for a scientifically relevant comparison, and I mean literally the same.

As an example of some metrics I would refer to the web since out there there's plenty of blog posts listing and comparing metrics. Just to link a few:

The list is not exhaustive but I think it illustrates the point.
Also, as a side note: almost all machine learning algorithms are optimization-based, if you want to refer to approaches that don't fall into machine learning I think a better term is analytic methods/approaches.

$\endgroup$
1
  • $\begingroup$ This is a great answer, I just want to suggest another side note. If OP is writing a scientific article, it might be worth adding another point in the Results and Discussion which is "how can the ML-based methods mimic the performance or compete with the performance of 'non-ML' based methods"? does it rediscover a certain (well-known) method? For example, an RL agent for travelling salesman problem could rediscover the greedy heuristic as shown by likelihood of taking the furthest node in each step. $\endgroup$
    – Sanyou
    Oct 5, 2021 at 8:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .