Seite - 151 - in The Future of Software Quality Assurance

Chasing Mutants 151 Thedegree towhichstatementsarecoveredbyanexecutable test suite isusually describedasstatementcoverage.Statementcoverageis thepercentageofexecutable statements thathavebeenexercisedbya test suite. StatementCoverage= NumberofExecutedStatements TotalStatements Toreachfull statementcoverageon thecodeaboveyouwouldneed two tests, as thereare twoexclusivepaths throughthecode required toexecuteeachstatement. Another coverage approach is to cover each branch or decision, essentially wherever a conditional statement such as if, for, while is included, ensure both outcomesof the conditional statement are evaluated.To reach full branchcoverage on the exampleabove,youwould needone further test, whichcoveredwhether the weekendvariablewas trueor false. BranchCoverage= NumberofExecutedBranches TotalBranches Finally, condition coverage, is a code coverage metric which measures whether each individualconditionhasbeenevaluated to trueor false.Thiscanbecalculated as: ConditionCoverage= NumberofExecutedOperands TotalOperands So referringagain to the exampleabove,ensuring that tests coverboth member- shipandaccesscardscenarios, addingonemore test to ourgrowingsuite. The problem with solely using code coverage metrics to measure the quality of automated tests is thatnoneof themetricsevaluatewhethermytests actuallycheck whether the customer is allowed access, or how much the software calculates as a charge.Theverificationof thesestatesandvariablesarenot includedin themetrics. Measuring code coverage on your automated tests is great, but it is only part of the picture. Code coverage only tells you the logic and branches which have been executed, it doesn’t really measure whetheryour tests are getting a lot of data of functional coverage, and it doesn’t tell you whether your tests are effectively detectingfailures. Validation of the system response (effectively comparing actual to expected results) is a critical part of implementing automated testing, it is straightforward to check that a variable is sensible; however, as interfacesget morecomplex (think of an XML message or a user interface) the amount of design subjectivity around thevalidations increases. Mutationoperationengines,whichapplymutationoperators tocode,vary in the operators they support, and indeed the user can usually configure how these are applied. As an illustration, the well-known Mothra study [4] and supporting tool used the operators on Fortran shown in Table 1. These are somewhat outdated, as operatorshaveevolvedwith thedevelopmentofobject-orientatedtechniques[5].Of