Page - 158 - in The Future of Software Quality Assurance

158 A. L.Smith Fuzz testing, which mutates the input domain of an application, can also be considered a type of mutation testing. Instead of mutating the program code, it mutates inputs to thesystemunder test.Thishascompletelydifferentgoals, though, to the mutation testingdescribedabove. It isprimarilyaimedatdetectinghowyour system will behave with unexpected inputs; this might be carried out as part of a security testing activity,oranegativefunctional test. Finally,mutationtestingcanbeused tounderstandthepropertiesofanunknown codebase: mutation generation, without running the kill phase, allows a nice and unbiased detection of complex code structures.—Markus Schirp This approach can be used to prioritize regression testing, or refactoring of a largecodebase. Whilemutation testinghasbeenaroundforawhile, it is solidlybuildingsupport in theengineeringcommunity,and it clearlydeliversuseful information it just isn’t possible togetelsewhere. 7 Conclusion This chapter has hopefully opened your eyes to a completely different approach to looking at test coverage. While mutation testing is something that is currently applied most frequently at the unit testing level, the concepts can be applied, and thebenefits realized throughouta full set of softwarequalityassurancepractices. It is not only an effective way to assess your automated tests, but also a way to understand the complexity of your code, and quantitatively understand your code qualityand test coverage. The concepts can also be applied to code, input data, the environment, and no doubtother technical artifacts. As explained, the biggest challenges are the compute resources required to execute a large number of tests on a large number of mutants, and also the humanresourcesrequiredtoanalyzeequivalentmutants,andsolveoracleproblems. Findingtherightbalancebetweencoverageconfidenceandresourcerequirementsis crucial.Limited researchexists, which covers the financial andqualitybenefitsand costs of mutation testing outside of research, and this is clearly an area that needs moreanalysis. Maybeitcan’thelpyourefactorallyourcodeor improveallyourtests,but itcan definitely point you in the right direction. No software quality assurance specialist needs less information, and no test suite can ever be fully understood in terms of coverage without some execution results. Mutation testing is no silver bullet, but cannotbe ignoredby truequalityprofessionals.