Seamless Multimodal Biometrics for Continuous Personalised Wellbeing Monitoring

Artificially intelligent perception is increasingly present in the lives of every one of us. Vehicles are no exception, as advanced driver assistance systems (ADAS) help us comply with speed limits, keep within the lanes, and avoid accidents. In the near future, pattern recognition will have an even stronger role in vehicles, as self-driving cars will require automated ways to understand what is happening around (and within) them and act accordingly. Within pattern recognition, biometrics offer promising applications in vehicles, from keyless access control to the automatic personalisation of driving and environmental conditions based on the recognised driver. Similarly, wellbeing monitoring technologies have long attracted attention to the possibility of recognising activity, emotions, sleepiness, or stress from drivers and passengers. However, these two topics are starkly opposed, since wellbeing recognition relies on intrasubject variability while biometrics thrives on intersubject variability. Despite their differences, biometric recognition and wellbeing monitoring could (and should) coexist. Continuous identity recognition from seamlessly acquired data could be used to personalise wellbeing monitoring models and attain improved performance. These personalised models could be the key to more robust ways of monitoring drivers? drowsiness and attention and avoiding accidents. In a broader sense, they could be applied to all vehicle occupants, paving the way towards the accurate recognition of activity, emotions, comfort, and even violence episodes in shared autonomous vehicles. This doctoral work focused on advancing in-vehicle sensing through the research of novel computer vision and pattern recognition methodologies for both biometrics and wellbeing monitoring. The main focus has been on electrocardiogram (ECG) biometrics, a trait well-known for its potential for seamless driver monitoring. Major efforts were devoted to achieving improved performance in identification and identity verification in off-the-person scenarios, well-known for increased noise and variability. Here, end-to-end deep learning ECG biometric solutions were proposed and important topics were addressed such as cross-database and long-term performance, waveform relevance through explainability, and interlead conversion. Face biometrics, a natural complement to the ECG in seamless unconstrained scenarios, was also studied in this work. The open challenges of masked face recognition and interpretability in biometrics were tackled in an effort to evolve towards algorithms that are more transparent, trustworthy, and robust to significant occlusions. Within the topic of wellbeing monitoring, improved solutions to multimodal emotion recognition in groups of people and activity/violence recognition in in-vehicle scenarios were proposed. At last, we also proposed a novel way to learn template security within end-to-end models, dismissing additional separate encryption processes, and a self-supervised learning approach tailored to sequential data, in order to ensure data security and optimal performance. Following the results of this work, one can conclude that truly personalised wellbeing is yet to be achieved. However, this work has built a strong framework to support future work towards the goal of integrating biometric recognition and wellbeing monitoring in a multimodal, seamless, continuous, and realistic way. Overall, this doctoral work led to numerous contributions to biometrics and wellbeing monitoring in general, resulting directly in twenty-four scientific publications in major biometrics and pattern recognition venues. Its quality and impact have been recognised by the scientific community with over three hundred citations and multiple awards, including the EAB Max Snijder Award 2022.