Contributions to Human Motion Modeling and Recognition using Non-intrusive Wearable Sensors

This thesis contributes to motion characterization through inertial and physiological signals captured by wearable devices and analyzed using signal processing and deep learning techniques. This research leverages the possibilities of motion analysis for three main applications: to know what physical activity a person is performing (Human Activity Recognition to identify who is performing that motion (user identification) or know how the movement is being performed (motor anomaly detection). Most previous research has addressed human motion modeling using invasive sensors in contact with the user or intrusive sensors that modify the user?s behavior while performing an action (cameras or microphones). In this sense, wearable devices such as smartphones and smartwatches can collect motion signals from users during their daily lives in a less invasive or intrusive way. Recently, there has been an exponential increase in research focused on inertial-signal processing to characterize people’s motion and develop systems with different applications. Machine learning systems are usually composed of a feature extraction module and a classifier module. The traditional systems typically try to extract handcrafted features based on expert knowledge from the signals. Then, the systems use these features as inputs to the machine learning algorithm to classify these signals into different classes. However, deep neural networks open the possibility to directly learn features from signals and perform both feature learning and classification tasks, through a unique architecture. These networks are very effective at extracting patterns when inputs highly depend on each other and can build models able to generalize complex patterns. Therefore, we decided to apply the learning capability of these networks for motion inertial signals based pattern modeling and recognition. Based on the hypothesis that each movement has special characteristics depending on its type of activity, we propose different signal processing and deep learning techniques on a typology of human activities: repetitive movements such as running or cycling; non-repetitive movements (gestures) as opening a drawer or drinking a cup of coffee; and postures such as sitting or standing. We have analyzed the window size, the signal analysis domain (time or frequency), and the neural network architecture to determine the most meaningful configuration for each type of activity. Long windows (> 25 s) of raw data in Convolutional Neural Networks (CNNs) provided better performance for repetitive movements. In the case of non-repetitive movements, the best option was using shorter windows (3 s) when modeling intra-window variations using Long Short-Term Memory (LSTM) layers (CNN+LSTM structure). For postures, detecting slow oscillations, thanks to a higher spectral resolution in long windows, allow increasing the recognition accuracy. Based on the hypothesis that the walking style can uniquely characterize a subject; we demonstrate that the learning capability of deep learning neural networks could competitively model identities to perform continuous person supervision. In this sense, we propose an adaptation of the d-vector approach used for the speaker recognition field and compare this method to traditional machine learning algorithms previously used in biometrics applications. The experiments based on the d-vectors solution have demonstrated the robustness of the proposed approach compared to the rest of conventional machine learning algorithms used in previous works. This analysis also includes the impact of different aspects such as the amount of enrollment time, recordings distribution in enrollment, validation and test subsets, and variability of considered activities. Regarding the third application, motor anomaly detection, we applied motion modeling in different scenarios to generate motion-based biomarkers. Firstly, for Parkinson?s Disease detection, we evaluated handwriting drawings as a non-intrusive procedure to remotely supervise the alteration in the kinematics of drawing through a simple digital tablet. We demonstrated that X and Y directions are the most informative signals for detecting Parkinson?s tremors. Secondly, regarding stress mood detection, physiological signals are crucial to evaluate stress situations and report quantitative metrics to physicians. Moreover, for detecting alcohol consumption, we investigated a motion biomarker through a wearable sensor demonstrating that intoxicated and sober subjects? movements could be modeled and classified using deep learning algorithms in a subject-dependent context. Finally, we demonstrated that estimating the reach distance of balance postural control exercises could be improved by applying recurrent networks. Finally, this thesis proposes several strategies for combining information from several sub-windows to improve the motion analysis or recognition in long durations. These strategies encompass feature learning and classification techniques since they are based on integrating information from motion in different points of the human motion modeling framework. Three alternatives for information combination have been described and evaluated: averaging the features before the deep learning algorithms, integrating the network outputs, or combining the information from several consecutive windows using deep learning structures like Time Distributed layers. As result of this analysis, we have demonstrated significant improvements when integrating information from sub-windows compared to directly using long windows. This work opens future lines to continue the research in the motion modeling field focused on improving activity recognition, biometrics, and motor anomalies detection with low intrusiveness in people?s day-to-day life.