Page - 148 - in The Future of Software Quality Assurance

148 A. L.Smith worse,inaproductionenvironment.Thisissomethingfamiliartoeverytestmanager astheydealwiththelearningfromproductionreleasesandfeedinginformationback intothetestprocess.Surelythereisabetterwaytofindproblemswithtestcoverage? MutationTestingisa testing techniquethatcanbetracedbackto1971[2],which has gained more and more attention, now with over a dozen associated tools and used in a range of software contexts. The number of tools available has grown significantly fromless than5 in1981, to over40 in 2013[3]. Mutation Testing sits in a gray area of testing techniques that don’t rely on formal specification of the desired behavior of the system, alongside fuzz testing, metamorphic testing, andarguablyexploratorytesting. At its core, mutation testing is the practice of executing a test, or a set of tests, over many versions of the software under test—a so-called mutant. Each version of the software under test has different faults deliberately and programmatically injected. Each testing iteration is conducted on a slightly different version of the software under test, with different faults injected based on heuristics, or “rules of thumb”thatcorrespondtocommonfaults.Theseversionsarereferredtoasmutants, in thecontext that theyare smallvariations .Thepurposeof the testing isusually to determinewhichfaults aredetectedby test procedures, resulting in failures. The manufactured faulty versions of the software aren’t called mutants because theyare inferior,butbecause theyarechangedin thesamewaythathumangenetics mutate as part of natural evolution. This is similar to the use of genetic algorithms inartificial intelligence to solvesearchandoptimizationproblems. Thebenefitsofmutationtestingcanbesignificant, it cangiveenormouslyuseful insight into: • Thequalityandcoverageofautomatedtests, inparticular,coverageofassertions andvalidations • Thecoverageof testing in aparticularareaof thesoftware • Thesuccessofparticular test design techniques • Thecomplexityandmaintainabilityofdifferentareasofcode • Thetraceabilityofsoftwarecodeorcomponentsto theoverallbusinessfunction- ality, throughthe tests It canbe usedasan approachto improveanexistingset of tests, or asa tool to help build a new test suite, either to ensure it has sufficient validations, or to prioritize testing towardsareasof thecode.Ultimately, it providesaset of factsaboutquality, information that isnototherwiseavailable (Fig.1). 2 Automated Testing MutationTesting conceptually is not limited to automated tests; inprinciple it is an approachthatcouldbeapplied tomanual testing.However, thecostassociatedwith running thousands of tests manually (on code that will never reach production, in order to improve the developmentand testing process) doesn’t stack up against the benefit inmostcases.