Page - 2 - in The Future of Software Quality Assurance

2 S. Amann and E. Jürgens In response to this fast-growing demand for testing, companies invest in test automation and continuous integration (CI) to speed up test execution. However, we increasingly see that even automated test suites run for multiple hours or even days, especially on larger systems. As a result, such long-running test suites are typically excluded from CI and executed only nightly, on weekends, or even less frequent.Asa result, the timebetweenthe introductionofamistake in thecodeand itsdetectiongrows.Thishassevereconsequences: • Largeamountsofchangespileupbetweentwoconsecutivetest runs, such that it becomesdifficult to identifywhichparticularchangecontains themistake. • Developers get feedback from tests only long after they did the changes that introduced the mistake, again making it more difficult for them to identify the mistake. • Theeffectsofmultiplemistakesmayoverlap,suchthatonemistakemayonlybe detectedafter another isfixed. Tomake thingsworse, test automationaddressesonlyhalf theproblem:While it improves the efficiency of test execution, it does nothing to ensure that the testing is effective. In practice, we see that about half of the changes may escape even rigorous testing processes [1, 2]. And as testers strive to make test suites more comprehensivebyaddingadditional tests, theyalsoadd to theruntimeof thesuites, thus, jeopardizing thebenefitsofautomated tests andCI. So howcan we break thisviciouscircleand makeour testingprocessesefficient and effective at the same time? The answer is surprisingly simple: We align our testingeffortswith thechanges.Withan increasingnumberofchanges tobe tested, it has become infeasible to simply execute all tests after every change and it has becomeimpractical—if it ever was—to manuallyensure that all changesare tested adequately. But instead of resolving to test only rarely, we should keep testing frequentlywithfocusonthechanges, i.e., thecodethatmightcontainnewmistakes. Wecall theideaofaligningourtestingwiththechangesinapplicationcodeChange- DrivenTesting.Change-DrivenTestingidentifies90%ofthemistakesthatourentire test suite may find in only 2% of the suite’s runtime [3] and informs us about any change to thecode thatwe didnot test [1,2]. 2 Test Intelligence To achieve high-quality testing, we commonly need to answer questions such as which test we need to run, what else we need to test, or whether our test suite contains redundant tests. Since it is difficult to correctly answer such questions manually, our goal is to automatically answer them using existing data from the software development process. This approach is similar to the approach of Business Intelligence, which analyzes readily available data to gain insights on businessprocesses.Therefore,werefertoourapproachasTestIntelligence.Figure1 illustrates it.