Page - 7 - in The Future of Software Quality Assurance

Change-Driven Testing 7 Relevant Changes Test-wise Coverage& Runtime Test-Impact Analysis Test Selection Test Prioritization ImpactedTests Orderedby Likelihood to Find a New Mistake Version-Control System Executionof All Tests Fig. 4 Process of the Test-Impact Analysis (TIA). Given a set of changes, TIA selects the impacted tests and orders them by their likelihood to find a new mistake in the changes to computea subset of the entire test suite that identifiesas many mistakesas early aspossible.Theprocessconsistsof twosteps: 1. In theTestSelectionstep,TIAselects the impactedtests, i.e.,all tests thatexecute anychangedmethod,accordingtotherecordedcoverage.Italsoincludesall tests that were added or modified by the change, because it cannot know which parts of the code those tests cover. 2. In theTestPrioritizationstep,TIAorderstheimpactedtestssuchthatallchanges are covered as quickly as possible, to find new mistakes as early as possible. Sincecomputingtheoptimalorderingofthetests is infeasible,TIAusesagreedy heuristic: It selects that testnext,whichcovers themostadditionalchangedcode perexecution time. In a study with twelve software systems [3] we found that the impacted tests selected by TIA find 99.3% of all randomly inserted mistakes that the entire test suite could find. The impacted tests found more than 90% of those mistakes in all studysystemsandeven100%in sevenof them. Inasecondstudywithover100differentsystems[4]wefoundthatTIAidentifies on average over 90% of the mistakes that the entire test suite identifies in only 2% of the execution time of the entire suite. Using TIA is especially beneficial for small and local changes, where test selection results in a small set of tests. This perfectly suits our need to quickly test many incoming changes, which are usually small compared to thecodeof theentire system. 3.2 Test-GapAnalysis Figure 5 depicts the process of the Test-Gap Analysis (TGA). It matches relevant code changes (see Sect.2.1) with the aggregated test coverage (see Sect.2.3) to identify those changes that were not covered by any test. We call these changes test gaps.