Seite - 155 - in The Future of Software Quality Assurance

Chasing Mutants 155 termsofcompilationperformance,butnobenefits in termsof theexecution timeof evaluatingeachmutant. Much research has been conducted into one concept called “Weak Mutation.” Thiseffectivelyevaluatesmutants at a component level, rather than fullyexecuting eachmutant.Thishasbeenfoundtobenearlyaseffectiveas“Strong”mutationwith reducedruntimecost [6]. Runtime The runtime cost of running mutation testing cycles can be significant. Of course, the execution can be scaled horizontally in most cases, as tests can be run across multiplemachines,or it can be scaled verticallyby usingmorepowerfulmachines. To apply horizontal scalability it is necessary to consider this when selecting the appropriatemutation testing tools,with support for suchapproaches. Classical testautomationtechniquesforimprovingtherun-timeoftestsalsoneed to be considered. Hard-coded waits built into tests may not be noticeable when the test is run once, but scaled to hundreds or thousands of executions, become a significant cost. Optimization of automated tests for performance purposes should beperformedbeforeattempting to introducemutation testing. Analysis Tworelatedareasofchallengeare theOracleProblem,andtheproblemofreducing equivalentmutants.TheOracleProblemis far fromunique to mutation testing, and applies to any area of testing where it is difficult to ascertain whether a test has succeeded. Thiscanoccurwhenmutantscan’tbekilled,becausetheassertionsorvalidations required to be implemented in the tests are toodifficult to implement. It can also occur when software is less deterministic, and when it is difficult to understand whether the failure to detect a mutation is actually meaningful. I asked AlexDenisov,authorofamutation testing toolcalledMull [7],whathe thought the biggestchallengeswere: As a user, the biggest issue so far is the test oracle problem. It is not a big deal on small or medium projects. However, for a big project, mutation can produce thousands of mutants, hundreds of which are surviving. It is not yet clear how a developer is supposed to process them—manually reviewing those hundreds is simply impractical.—AlexDenisov Theproblemofreducingequivalentmutants isalso relevant, that is, themutation does not result in observable change to the output or execution. This can occur because of dead code that isn’t used; the mutation only changes the speed of the software;or, the mutationonlychanges internallyuseddata that doesnot affect the endstate oroutput.