Page - 5 - in The Future of Software Quality Assurance

Change-Driven Testing 5 cangroupallchanges thatbelongto thesameticket.Thisallowsus tofocusononly thechangesmotivatedbyaparticular ticket. Figure2cshowsthechangesrelatedtoasingle ticketona treemap.Methodsthat were added by a change are highlighted in red and methods that were changed are highlighted inyellow. 2.3 Profilers Profilers, such as Jacoco or gcov, record which parts of the application code are executed, i.e., they record code coverage. Dependingon the technology in use, profiling approaches range from instrumenting the target code, over attaching a profiler to a virtual machine, to using hardware profilers. Regardless, profiling is alwayspossible. Different profilers record coverage at different granularity, e.g., they record which methods, statements, or branches get executed. With most profilers, a finer granularity results in a larger performancepenalty. Recording coverage on method level is, inourexperience,always feasiblewithanacceptableoverhead. When a profiler records the coverage produced by a test, we speak of test coverage. Recording test coverage is a widely used practice for automated tests in CI and uncommonin other typesof test environments.Technically,however,we may profile any type of execution, be it through a unit test or an end-to-end test, automated or manual. Thus, using a profiler, we may obtain the coverage of each test inourentire test suite. Existingprofilerstypicallyaggregatethecoverageofall testsexecutedinasingle test run, because they work independently of the test controller and simply trace execution from start to end of the run. However, conceptually,we may also record test-wise coverage, i.e., separate coverage for each test case. When we record test- wise coverage,we mayalso trivially recordeach test’s individual runtime. Figure 2d shows the aggregated coverage of a test suite on a treemap. Methods thatwereexecutedare highlighted ingreenandmethods thatwerenotexecutedare highlighted ingray. 3 Change-Driven Testing Change-Driven Testing is one particular instance of Test Intelligence that makes testing both more efficient and effective. Figure 3 depicts the idea. The key insight behind Change-Driven Testing is that existing tests will only fail if new mistakes are introduced,1 and that new mistakes can only be introduced through changes. Consequently, when considering the (possibly buggy) changes between 1Leaving aside causes for sporadic test failures, such as flaky tests or interaction with third-party systems, which typically indicate a problem with the test setup rather than with the system under test.