Page - 13 - in The Future of Software Quality Assurance

Change-Driven Testing 13 5 AdaptingChange-Driven Testing In practice, we encountervery different testing setups and strategies, dependingon the concrete requirementsand the history of the respective projects. Consequently, theimplementationofChange-DrivenTestingshouldbeadjustedtodeliverthemost valuegiventheproject’sparameters.Wesubsequentlydiscusssomeaspects thatwe encounter repeatedly,withoutaiming foranexhaustive list. • Both Test-Impact Analysis and Test-Gap Analysis may consider test coverage from all testing stages as well as from a combination of both automated and manual tests. The type of test literally makes no difference for the benefits of Test-Gap Analysis. Test-Impact Analysis, on the other hand, is especially beneficial if tests are time consuming, i.e., if excluding tests saves significant time,and if testing time is limited,because testprioritizationmakes it likely that we catchmistakesearly,even ifwe cannotafford to runall impacted tests. • If developmentmostly happenson feature branches,a sensible strategy is to use Test-ImpactAnalysisfortestingchangesandticketcoveragetoavoidtestgapson thesebranches.Inaddition,toensurenomistakesslipthroughtesting,thefull test suite should be executed upon merge of a feature branch. If, on the other hand, developmenthappensonamainbranch,wemayuseTest-ImpactAnalysis to test individualchangesand run the full test suite periodicallyorbeforea release. • It is always possible to combine Test-Impact Analysis with other test-selection strategies, e.g., if some tests for highly critical functionality should always run. We thensimplyrun theunionof the impacted tests selectedbyTIAand the tests identifiedbyanycomplementarystrategy. • We found that it is most efficient to investigate test gaps using ticket coverage, because the ticket provides us with additional context when analyzing the gaps. However,evenifTGAcannotmapcodechangesto tickets, it canreveal testgaps for the system as a whole. Such a system-wide TGA is valuable on its own as muchas inaddition to ticket coverage,asa second-levelqualitygate. • In many projects, the people analyzing test gaps are not necessarily the devel- opers who wrote the code changes, but testers or architects. Such a separation of work sometimes makes the interpretation of test gaps more difficult, because the analysts may be unaware of possible reasons for a particular change or test gap. Insuchcases, thedatafromtheversion-controlsystemagainproveshelpful, because it names the developer responsible for any change, telling the analysts who to talk to. • While it is the theoretical ideal, it is never a goal in itself to reach an all-green treemap, i.e., 100% test coverage. In many situations, leaving test gaps is quite reasonable, e.g., if the gap is on code that prepares future functionality, but that it not yet live or if it is on code that is only rarely used internally, such that testing resources are better invested elsewhere. In the end, the testing process is alwayssubject to tradeoffsandprioritization.TGAenablesus tomakeconscious decisionsaboutwhere todirectour limited resources.