5
$\begingroup$

This is my first project using machine learning so I'm looking for some guidance. I am extending a model-based testing (MBT) system to a learning-based testing system by integrating a machine learning algorithm, for automation purposes. The MBT system executes tests on some system under test (SUT) and generates test verdicts.

The ML algorithm I want to integrate, is to take test verdicts from the MBT system as input and gain understanding of the SUT's behaviour, in order to generate new test cases. The test verdict (input) is a text-file, containing information about the previous test and whether it passed or failed. The output is a new test case, also a text file, containing variable values.

I was thinking that supervised learning would be suitable since the input file contains both variable values (features) and the test verdict (class). However, I have doubts since I am not looking to solve a classification problem.

I would appreciate ideas of what type of algorithm I should use (supervised/unsupervised/reinforcement etc.), and where I could find such (open-source) algorithms.

Thanks,

Adam

$\endgroup$
  • $\begingroup$ What is the purpose/goal of generating the new test cases? Are you looking for variations of parameters to the SUT that might expose flaws? Is there a human expert version of this process - i.e. working purely from the test verdict (and no other data about the SUT), an expert can determine that a new test is required and what the test's parameters should be? $\endgroup$ – Neil Slater Mar 20 at 10:13
  • $\begingroup$ Hi @NeilSlater, thank you for your answer! The purpose of generating new test cases is to expose flaws in the SUT. The system is to be fully automated and work from test verdicts. For reference a timed automata (TA) model of the SUT is provided, expressed in NuSMV. This is in order to have a reference of the desired behaviour of the SUT (e.g. timing requirements between states, to see if the SUT violates them). There will be no human tester who determines what is tested, rather the aim is for the algorithm to determine that on its own, based on previous tests. $\endgroup$ – putin putout Mar 20 at 10:55
  • 1
    $\begingroup$ I understand that you don't want a human tester in the loop when this system runs. However, to assess whether an AI could do what you want, a useful reference is "could a human do this task, given the same information?" Could you answer that, please, as it is a useful check on whether your idea is feasible. Also relevant is "How would you know whether a human (or any) worker doing this task has done a good job?" $\endgroup$ – Neil Slater Mar 20 at 13:03
  • $\begingroup$ For example I could generate millions of test cases very easily by choosing parameters randomly. I presume that is not a valid goal, and I would be a bad human expert at this task (do note I am developer and spend a significant amount of my time writing automated tests - but I select those tests based on knowledge of the system and "white box" knowledge of the SUT). Do you have any way of expressing the utility of a new test case? $\endgroup$ – Neil Slater Mar 20 at 13:06
  • $\begingroup$ @NeilSlater I see. Yes, a human can do the task given the same information. The ML algorithm is provided with a Timed Automata (TA) model of the SUT, which gives the white box knowledge of the SUT and explains boundaries on parameters. The new test case should test different aspects of the system than the previous test cases, however within the limits of the system's abilities, explained by the TA model. I hope that answers your question. $\endgroup$ – putin putout Mar 21 at 8:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.