Page - 203 - in Proceedings - OAGM & ARW Joint Workshop 2016 on "Computer Vision and Robotics“

Figure 2: System architecture 2.1. Skeletal Joints Estimation Estimation of skeletal joints is a crucial pre-requisite to overcome real-time data loss. The sampling rate of currently affordable RGB-D sensors is 30 fps. Recent works [3, Section 1.1], [9, Section 1.1] indicates that this sampling rate is not sufficient to recognize human activity in less than 1s. This leads to the motivation of estimating the skeletal joints data with a higher sampling rate. In the first stage, mathematical modelling of skeletal joints of left and right hands with respect to Head, Neck and Spine Shoulder skeletal joints will be performed in offline. In the second stage, the measured skeletal joints will be fed to a zero order hold (ZOH) component to provide the k th sample at time instant k*Ts with repeated values until the k+1 th sample appears at time instant (k+1)*Ts. To overcome real-time data loss at time instant k*Ts, extrapolated values for skeletal joints of the left and right hands will be generated from the mathematical model. In the third stage, the samples with the higher sampling rate resulting from the ZOH and the extrapolated values resulting from the mathematical model will be used for estimating the desired skeletal joints positions. A forward Markov model describing the desired skeletal joints positions will be assumed and a stochastic subspace realization algorithm [8] will be applied to estimate the desired skeletal joint positions. 2.2. Activity and Task Recognition Activity is defined as the sequence of actions or a single action performed by a human and his/her interactions with the objects of interest within an arbitrarily short time window. During the offline stage, probabilities of the recognized actions, human-object interactions and actual positions of robot’s joints are considered as activity specific features and are collected with respect to M activity demonstrations by L individuals. Here, human-object interactions are represented by human motion trajectories and 3D position information, IDs and probability values of tracked objects. The recorded M*L demonstrations are then fed to a classifier for activity classification. A Markov model will be adopted to represent the temporal relationship between human activities over time. During the online stage, partial segment of the activity specific features are used as inputs to compute the probability for states which represents human activities. The state with the highest probability will then be the recognized activity [12]. The activity recognition approach mentioned in this section will be extended for task recognition using a Hidden Markov model (HMM) to represent the process steps/task as its states. In the case of task recognition, the probability values of human 203