Page - 153 - in The Future of Software Quality Assurance

Chasing Mutants 153 In this example, the red items will be changed. The first will be changed to be initializedas true, the secondwill changeanor toan and, the thirdand fourthwill negateaBooleanvaluebyaddinganot. As a result, four versions of the system under test will be compiled, and the mutationtestingroutineshouldrunallautomatedtestsagainsteachversion.Inorder to successfully detect each of the mutants, the test suite would need the following characteristics: • To test that the customer was not permitted access if the customer did not have either card, and it was not a weekend. This would test a failure to initialize the variable in mutation1. • To test wherea customerhasonecard,butnot theother (full conditioncoverage wouldalso require this), and thenvalidate thecustomerwaspermittedaccess. • To test that a customerwasallowedaccesswithoutcardsona weekend. • Replacingvariablevalues As you can see, although a test suite built to achieve code coverage would exercise similar paths through the code, mutation testing metrics allow much more specificityabout theverification the tests shouldperform. This isusefulbecause,unsurprisingly,manysoftwarefaultsare introducedin the coding process. For example, the common off-by-one error, where a programmer instructsa looptoiterateonetimetomany,ortoofew—ormiscalculatesaboundary condition—is directly addressed by test design techniques such as boundary value analysis and equivalence partitioning. Similarly, this kind of error is commonly injectedbymutation test operators. Themutation testingprocesscanbesummarizedas follows: 1. First, mutantsarecreated,by insertingerrors. 2. After theyhavebeencreated, the tests are selectedandexecuted. 3. Themutantswill be“killed” if the tests failwhenexecutedagainst themutant. 4. If theresultof testingthemutantis thesameaswiththebasesoftware, themutant has“survived.” 5. New tests can be added, existing tests amended, or code refactored—in order to increase the number of “killed” mutants. Some mutations cannot ever be detected, as they produce equivalent output to the original software under test, these arecalled“equivalentmutants.” Oncethefullprocesshasbeenexecuted,amutationscorecanbecalculated.This is the ratioofkilledmutants, to the totalnumberofmutants.Thecloser thisscore is to1, thehigher thequalityof the testingsuite and the software. MutationScore= NumberofKilled Mutants TotalMutants