Page - 157 - in The Future of Software Quality Assurance

Chasing Mutants 157 However it isn’t just thecoremutationengine thatyouneed toconsider: As an implementor, the bigger problem I see is the smooth integration. How to include the mutation testing into existing infrastructure: various build systems, test frameworks, CI pipelines, IDEs, etc.—AlexDenisov There are other types of tools that you might want to look at with regard to youroverallworkflow.For instance,Pitest, a Javamutationengine,hasaCucumber plugin [9] that allows you to integrate with Cucumber and Behavior-Driven Development.Similarly,SonarQube[10], thepopularcodequalitymonitoringsuite, has plugins that allow you to display detailed results from your mutation test run. Some mutation testing engines also offer IDE plugins that accelerate the feedback loopandallowyouto see the results in the contextofyourcode. 6 OtherApplications ofMutation Testing Onewaytolookatscalingmutationtestingis in thetest level. It isfareasier toapply mutation testing at a unit test level, because individualmutationscan be intuitively tracked to effects in the software’s operation, less resources are required, and the start-upandexecution timeforeach test ismuchsmaller, leading to faster results. Mutationtestingcanstillbeappliedathighertest levelssuchasfunctionalsystem testing. Mutated software versions can be deployed to web servers, and typical tools like Selenium can be used to run the test iterations. However the execution time, and amount of setup work, will be considerably higher. Investigating and resolving issuescanalso requiremoreworkto link themutated lineofcodeback to a functional test verification. The concept of mutation testing has been proposed to cover other areas of software quality as well. For example, it has been proposed, at least in research [11], that mutation can be used on top of formal specification languages to detect defectsat the designstage. Mutation testing can also be used to inject faults in the running environment. Chaos Monkey, a tool originally developed by Netflix [12] to test the resilience of theirenvironments,randomlyinjects infrastructurefaults, toseehowtheapplication under test (or live application!)handles the failure. This can be viewed asmutating theenvironment thesoftware runs in: Failures happen, and they inevitably happen when least desired. If your application can’t tolerate a system failure would you rather find out by being paged at 3am or after you are in the office having already had your morning coffee?—Chaos Monkey on github