Page - 154 - in The Future of Software Quality Assurance

154 A. L.Smith 4 Mutation TestingChallenges andStrategies Mutationtesting is inherentlyscalableas it isusuallybasedondoingsomethingyou alreadydoautomatically,manytimesoverandindependently.Thatsaid, thisshould alsobe trueofautomated integrationandsystemtesting,but isoftennot. The resources required to conduct mutation testing can be significant, clogging up the continuous integrationpipeline forhours to runendless iterationsof tests on endless iterationsofmutatedsoftware.Additionally the timerequiredtogothrough andfix all the issuesuncovered,canbedaunting. Threedistinct areasofcost need to beconsidered: • Thecompile-timecostofgeneratingmutants • The run-timecostof runningtests on themutants • Thehumancostofanalyzingresults CompileTime As mentioned previously, a major problem with mutation testing is the cost of execution. The number of mutants is a product of the number of lines of code and thenumberofdataobjects, but asa ruleof thumbthenumberofgeneratedmutants is typically in theorderof thesquareof thenumberof linesofcode.Somestrategies [6]havebeen tried to reduce theamountofexecution: • Sampling—Executingonly a randomsample of mutants across a logical area of softwareandassociated tests. • Clustering—Usingunsupervisedmachinelearningalgorithms(e.g.,K-means) to select mutants. • Selective Testing—Reducing the number of mutation operators, that is, the heuristicsused to inject faults, can reduce thenumberofmutantsby60 • Higher-order Mutation—First-order mutants are those that have had a single fault injection; second-order mutants have been injected with multiple faults, in multiple iterations of mutation. Higher-order mutants are harder to kill, and focusing solely on second-order mutants has been shown to lead to reduced effort,without reducingcoverage. • Incremental Mutation—Mutating only new or changed code, rather than the wholecodebaseunder test. We know that defects cluster in the same areas of code, and perhaps a simple strategyofapplyingthetechniquestolimited,complex,high-risk,anddefect-ridden areasof functionalitycanofferanappropriatebalanceofcost vsbenefit. Conversely, there are statements in code that we not be concerned about. For example, excluding all logging statements from mutation may be appropriate and lead to less mutants to test. Another approach used to reduce the amount of time and resources required is direct integration with the compiler. Early mutation testing approaches compiled each mutant individually; however, more modern approaches compile once, and then mutatean intermediary formsuch as bytecode.This has significant benefits in