Page - 149 - in The Future of Software Quality Assurance

Chasing Mutants 149 Software Under Test 100% Tests Pass Mutant 1 Mutant 2 Mutant 3 95% Tests Pass90% Tests Pass100% Tests Pass mutation operationmutation operation Fig. 1 In thisexample,Mutant1doesn’t cause any tests topass, indicating that the injected faults isnotdetected,hence there isa fault in thetestcoverage.Mutants2and3cause test failures,which isgood! One of the core challenges with automated testing is ensuring that there is the rightbalanceofassertionsandvalidationchecks toensure that software isworking, withoutmaking the test suite brittle to irrelevant structural changes in the software. Althoughusuallyslower,weknowthathumantestersfindmoredefects insoftware. We know that the reason for this is that the human tester has more context, and uses experience and heuristics to determine correct behavior as well as the stated assertions in any testing procedure. Unlike with manual testing, it is a given that automated test suites are by nature, only going to detect defects within the bounds of the validations they have been setup to perform—in other words, they will only check the things theyhavebeen told tocheck. In themost extremeexample, it ispossible to createanautomated test that starts the software under test and validates it appears to execute successfully, without makinganyassertionsabout theprogramoutput.Thiswouldbeaverymaintainable test, as it would rarely need to be updated to stay in line with the software, but a very low value test, as it tells us little other than that the program executes. If this hypotheticalsoftwareandtestwere tobeevaluatedusingmutationtesting, thescore would be very low, as mutated versions of the software would only fail the testing if the injected fault causeda full executionfailure. Taking a counter-example, an automated test against a website that compares every single accessible element of the HTTP responses against a baseline copy. This would be a very unstable test, as it would fail at even the smallest change, perhapsonethatarealuserortestercouldnotperceive.If thissoftwareandtestwere evaluatedwithmutationtesting, thescorewouldbeextremelyhigh,asvirtuallyany changewouldcausea test failure. Mutation testing does not solve the problem of the maintainability of test automation, but it does give useful insight into the value actually offered by individual tests. This is also important because even in an environment with a high level of automation,exhaustiveor full combinatorial testing is oftennotperformedbecause it is consideredan impracticaluseof resources. One risk with any test design technique is test case explosion—that is, when you focus on a specific test design technique, and the volume of tests dramatically