Seite - 9 - in The Future of Software Quality Assurance

Change-Driven Testing 9 capturetheimpactofthetestand,therefore,cannotadequatelyselect impactedtests. In practice, even the coverage of automated tests may vary between runs, e.g., due to garbage collection. Though the effects are typically small, this means that TIA cannot guarantee that the impacted tests always find all the mistakes that the entire test suite may find. To the best of ourknowledge, there is no test-selection strategy that gives such a guarantee under these conditions. Therefore, we recommend to regularly execute the entire test suite to ensure that nothing slipped through and to keep the data about test runtimes and coverage up to date. This is the same as using a traditional regular (e.g., nightly) testing strategy, except that we now have the additional fast feedback from the immediate testing using TIA, which already captures themajorityofmistakes. Another limitation of TGA is that it doesnot considerhow thoroughlya change wastested,butonlywhetherthemethodscontainingthechangewereatallexecuted in a test. Like any other testing approach, TGA cannot prove the absence of mistakes in the code. However, by revealing changes that have not been tested at all, it identifies changes for which we are certain that no mistakes can have been found. Such untested changes are five times more likely to contain mistakes [2]. In our experience, TGA usually reveals substantial test gaps and, thereby, enables significant improvementsof the testingprocess. 4 UsingChange-Driven Testing To illustrate the use of Change-Driven Testing, we follow the development of a feature of our own software productTeamscale.Teamscale is a platform that assists softwaredevelopmentteamsin theanalysis,monitoring,andoptimizationof codeand test quality.For thispurpose, it integrateswith variousotherdevelopment tools, such as version-control systems, build servers, and ticket systems. Its web frontend provides an administration perspective that allows to manage credentials for suchexternal systems. Figure6shows feature requestTS-15717 that asked to add the possibility to delete suchexternalcredentials. Before starting to work onTS-15717, our developer creates a feature branch toworkon.Thenshe implements the relativelysmall feature,writesanewtest, and commits thechangesas TS-15717: Support deletion ofexternal credentials. The treemapin Fig.2cshowsallof these codechanges. 4.1 TestingwithTIA Triggered by the commit, our CI environment starts building Teamscale and testing the change. In the testing stage, the CI queries TIA to learn which tests toexecute.Tocomputeits response,TIAuses the test-wise runtimeandcoverageof