Page - 8 - in The Future of Software Quality Assurance

8 S. Amann and E. Jürgens Relevant Changes Accumulated Test Coverage Test Gaps Test-Gap Analysis Execution of All Tests Version-Control System Fig. 5 Processof theTest-GapAnalysis (TGA).FromagivensetofchangesTGAidentifies those changes thatwere not yet tested, i.e., test gaps To determine test gaps, TGA considers the chronological order of changes and test runs: New changes invalidate any previous test coverage of the changed code and open new test gaps. With subsequent testing, new coverage is recorded, which closes respective test gaps on earlier changes. Consequently, TGA can give us an update on our test gaps after every test run and every code change. And since the analysis works incrementally, it computes the update in a matter of seconds, even forvery largesystems. Inacasestudyona large industrial softwaresystem[1]TGArevealed thatmore than 55% of the code changes in two consecutive releases remained untested. In retrospect,we tracedover70%of the reportedfieldbugsback tountestedcode. In a second case study on another industrial software system [5] TGA found 110 test gaps in the changes from 54 tickets, which corresponds to 21% of the 511 change methods. Provided with the data, developers found 35% of these gaps worth testing. Interestingly, they also found that 49% of the gaps resulted from cleanupworkthatwasnotpartof the ticketdescriptionand, thus, remaineduntested evenafter thechangerequestedby the ticketwas thoroughlytested. 3.3 Limitations Tomakebestuseoftheanalyses, it is importanttobeawareof their limitations.One limitationofbothTIAandTGAischangeson theconfigurationordata level.Since such changes are not reflected in the code, they remain hidden from the analyses. Consequently, TIA cannot adequately select impacted test and TGA cannot show impactedpartsof thecodeasuntested. A related limitation of TIA comes from the use of indirect calls. For example, if a test executesall classes with a certain annotation,TIA typically doesnot select this test when the annotation is added to another class, because the test’s historical impactdidnot include thatclassand the test itselfdidnotchangeon thecode level. TIAassumesthat thecoverageof testcases isstable, i.e., thatexecutingthesame test twice results in the same coverage. If this is not the case, e.g., because manual test steps are performed differently with every execution, TIA cannot properly