Page - 69 - in The Future of Software Quality Assurance

Testing Autonomous Systems 69 requirements the system to be tested should meet and what should therefore be checked indetailby test cases. Utilizing this approach a test plan for the mobile robot “Mobipick” [14] of the German Research Center for Artificial Intelligence (DFKI) was created in 2018 as part of a cooperationproject between imbus AG and DFKI. The test contentswere recorded in the cloud-based test management system [15] and made available to the DFKI scientists and the project team. The following list references this case studyto illustratewhich topicsandquestionsneedtobeconsideredwhentestingan autonomoussystem: • Functional Suitability: It must be checked if the functional properties of the systemare implemented“complete,”“correct,”and“appropriate.”Thefunctions of each individual component of the system are affected (at lower levels). At the highest level, the ability of the overall system to complete its mission shall be tested. The “Mobipick” test cases for example focuses on the functions “Navigation” and “Grabbing” and the resulting mission pattern: approach an object at a destination, grab it, pick it up and put it down at another location. Testing the functionality also must include testing the system’s load limits! A restriction forgrippingcouldbe, forexample, that the robot tipsover in the case of heavyobjectsor is deflected fromits directionof travel.Such boundarycases and the system behavior in such boundary cases must also be considered and examined. • PerformanceEfficiency:Thetimebehaviorof thesystemanditscomponentsand the consumptionof resourcesmust bechecked. – Possible questions regarding time behavior are: is the exercise of a function (e.g., obstacle detection) or mission (object approach, grab, and pick up) expected in acertain timeperiodorwith acertain (min/max)speed? – Possible tests regarding resource consumption (e.g., battery power) are: run longest application scenario on full battery to check range of battery; start mission on low battery to check out of energybehavior; start mission on low battery at differentdistances to chargingstation to check station location and estimatepowerconsumptionto station. • Compatibility: This concerns the interoperability between components of the system itself (sensors, controls, actuators) as well as compatibilitywith external systems. Possible questionsare: Can the control software, whichwas brought to the robot initiallyorafter anupdate, takeoversensordata,process it andcontrol actuators correctly? Are the protocols for communication compatible between robotcomponentsorwith external systems? • Usability: What possibilities does the user have to operate the robot or to communicatewith it?Howareordersgivento therobot?Are thereanyfeedback messagesintheeventofoperatingerrorsandfailuretounderstandthecommand? Howdoes therobotcommunicate its status?Whichchannelsandmediaareused for transmission:via touchpanelon the robot,via appoverWLAN, orviavoice control? This also includes the handling of objects: can the robot hand over a grippedobject to itsuserpreciselyenough?