The testing problem in traditional software has been fully explored over the last decades, but it seems that testing in artificial intelligence/machine learning has not (see this question and this one).

What are the differences between the two?

  • $\begingroup$ When you say "The testing problem in traditional machine learning", maybe you meant "The testing problem in traditional software"? Moreover, please, clarify what you mean by "testing" because people could confuse "testing" with the typical testing phase of machine learning, where you test the generalization of the model, which is not what I think what you mean necessarily. $\endgroup$ – nbro Apr 5 at 15:31
  • $\begingroup$ @nbro Sorry for the typo. $\endgroup$ – Lerner Zhang Apr 5 at 15:33

Testing machine learning programs is quite different than testing traditional software.

The main reason why this is the case is quite simple, if you're familiar with machine learning.

ML programs are not just if statements and loops, but they are composed of models, which can even be black-box models, such as neural networks (i.e. it's difficult to interpret the function that they compute). These models are trained to approximate some unknown function, which is only partially described by some given data, which can also contain spurious/noisy information.

For this reason, traditional testing techniques, such as statement coverage, are insufficient to fully "test" ML programs: e.g. even if all statements of your ML program are covered, your ML program can still fail, e.g. it may predict the wrong class for some previously unseen input. So, testing ML programs requires not only traditional testing techniques, but also other approaches that attempt to address the (un-)desirable behaviour (such as the generalization) of the models.

Currently, I am also doing research on this specific topic, so I can say that there are already several people working on this topic, but the field is still quite immature and there are still many unsolved problems. It's also important to note that the general "testing problem", even for traditional software, is not yet solved.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.