I'm trying to develop a kind of AI that will assist in debugging a large software system while we run test cases on it. The AI can get access to anything that the developers of the system can, namely log files, and execution data from trace points. It also has access to the structure of the system, and all of the source code.
The end goal of this AI is to be able to detect runtime errors during execution, and locate the source of these errors.
I was considering making use of a deep neural network, where the input would be the execution data and log output. Using this input it would be able to verify whether the current version of the system we are running is functional, or non-functional. The problems with this approach is that the system it would be evaluating would be constantly changing as it gets developed, so the only training material the NN would have is from the last stable version of the system (and even that could have some errors). Additionally, producing test cases for the system off of which we could train the NN would be very time consuming, and would defeat the purpose of using the NN in the first place.
I would like to know what AI design you think would be suitable for this task. Please let me know if you would like any other information relevant to the problem. As far as I can tell, nothing quite like this has been done before.
It's probably worth mentioning that my team has a some extremely powerful machines on which we can run the AI.