Page - 73 - in The Future of Software Quality Assurance

Testing Autonomous Systems 73 scenarios, it isdifficultandriskytoreducetestingeffortbyprioritizingandomitting scenarios. Testing only one such scenario in reality can require enormous effort (a secure test site is required, and changing the test setup and the subsequent repeated test drives in that site requires a lot of effort and time). A very large proportion of the necessary tests mustandwill thereforebecarriedout in the formofsimulations. Nevertheless, some of the scenarios will always have to take place additionally in reality. Because simulations can be error-prone and they usually will not be physicallycomplete. An importantmeasure to gain time andsafety is a consistent shift-left of tests to the lowest possible test levels andcontinuous testing duringdevelopmentat all test levels in parallel: at the level of each individual component, for each subsystem, and at the system level. Test-driven development and the formal verification of safety-critical components will play an increasingly important role. Continuous monitoring of the systems in operation (“shift-right”) and, if necessary, quick reactiontoproblemsin thefield,will alsobeindispensable.In theEthicsGuidelines forTrustworthyAI of theEuropeanCommissioncorrespondingdemandsareclearly formulated:“Testingandvalidationof thesystemshouldoccurasearlyaspossible, ensuring that the system behaves as intended throughout its entire life cycle and especially after deployment. It should include all components of an AI system, including data, pre-trained models, environments and the behaviour of the system asawhole.” [3]. Thetestcontentsandtest resultsofall test levelsandthedatafromfleetoperation must be continuously monitored, evaluated, and checked by test management in ordertobeable to identifygapsinthe testcoveragebutalso toreduceredundancies. Significantly increased importance will be attached to testing by independent third parties. Here, too, [3] formulates proposals: “The testing processes should be designed and performed by an as diverse group of people as possible. Multiple metricsshouldbedevelopedtocoverthecategoriesthatarebeingtestedfordifferent perspectives. Adversarial testing by trusted and diverse ‘red teams’ deliberately attempting to ‘break’ the system to find vulnerabilities, and ‘bug bounties’ that incentiviseoutsiders todetectandresponsiblyreportsystemerrorsandweaknesses, canbeconsidered.”. 5 ConclusionandOutlook Proceduresandbestpractices fromthe testingofclassical softwareandITsystems, as well as from the field of conventional, safety-critical systems or vehicle compo- nents,14 arealsostill valid for the testing ofautonomoussystems. 14ISO 26262:2018, “Road vehicles - Functional safety,” is the ISO series of standards for safety- related electrical/electronic systems inmotor vehicles.