One of the issues in testing many applications and software systems is time to execute adequate combinatorics. I propose that the possibility exists to create a system that could simultaneously test all the most important combinatorics in an inexpensive massively parallel processing system.
Requirements: The system would have to be affordable. It would have to run an open source or other adaptable OS. There would have to be integration between the testing software and the operating system.
Hardware: Parallella enables massively parallel systems inexpensively. It's an open source system, which would allow it to be customized to the needs of such a testing environment. I should state here that I am not experienced with this system.
Low level software: Low level calls would have to be provided which give testing tools the ability to branch an execution, such that one code path continues along decision path A, while another is spawned that follows decision path B. The testing tools would also require services that let them identify these branchings for reporting.
Testing Software: The testing software itself would have to be able to trigger these branchings, and to track on and report their progress. As well as to fail out gracefully and with logging in the event of an error. I would propose that these services be created as the lower level, and provided so many competing tools could be created for different needs.
Benefits and Limitations: Even with such a system, not all systems would be applicable to this kind of testing, and for most systems there may well be too many combinatorics to test them all for a given limit on cost of hardware. However by creating systems targeted to the most important subset of combinatorics, such a system could provide much more comprehensive testing for a given amount of time than existing solutions.
In Summary: While such a system would have limitations, it could be built inexpensively, with existing components in an open source manner. This would create a broadly applicable and inexpensive solution for the projects it was suited for. In short, once built, a lot of benefit for a remarkably small cost. And a technically interesting problem as well.
I dont know if i am saying the same thing in different language, but If you run the software in a virtual machine, then every time you reach a branch point you can copy the memory space into 2 or more VMs and continue testing each alternative without having to recover the initial code.
ReplyDeleteYou have effectively simulated the quantum physics many parallel universes interpretation.
Now if you also check off what memory was touched in each branch and which memory is used in branching decisions you can determine whether spawning another branch is necessary when code contains loops. This way you could exhaustively test a chess program even without testing all possible positions.
For high combinatoric branchings you can determine what the probability is that a user will enter one of the as yet untested branches. You can look at the bug density found in tested branches, and the unit cost of testing a branch and trigger your test framework to stop testing when the expected probability of failure is less than the cost effective reliability threshold you determine.
Finally, if you work in collaboration with customers, you can mark code in the field so that it detects untested branches when they are entered. If the branch is successfully exited it has now been user field tested and you can flow this data back to the central test site. QA can begin testing other branches that spawn from the same initial point where the user exited the Pre-release testing. This can guide testing based on most likely field conditions and allow you to enlarge the tested space in a way that minimizes user exits over time. In effect you are re tagging probabilities in the untested space based on what users actually do, so you get Bayesian probabilities not mere combinatoric probabilities.
Scott McGregor