Facial Soft Biometrics: Methods, Applications and Solutions

This dissertation studies soft biometrics traits, their applicability in different security and commercial scenarios, as well as related usability aspects. We place the emphasis on human facial soft biometric traits which constitute the set of physical, adhered or behavioral human characteristics that can partially differentiate, classify and identify humans. Such traits, which include characteristics like age, gender, skin and eye color, the presence of glasses, moustache or beard, inherit several advantages such as ease of acquisition, as well as a natural compatibility with how humans perceive their surroundings. Specifically, soft biometric traits are compatible with the human process of classifying and recalling our environment, a process which involves constructions of hierarchical structures of different refined traits. This thesis explores these traits, and their application in soft biometric systems (SBSs and specifically focuses on how such systems can achieve different goals including database search pruning, human identification, human re?identification and, on a different note, prediction and quantification of facial aesthetics. Our motivation originates from the emerging importance of such applications in our evolving society, as well as from the practicality of such systems. SBSs generally benefit from the non-intrusive nature of acquiring soft biometric traits, and enjoy computational efficiency which in turn allows for fast, enrolment?free and pose?flexible biometric analysis, even in the absence of consent and cooperation by the involved human subjects. These benefits render soft biometrics indispensable in applications that involve processing of real life images and videos. In terms of security, we focus on three novel functionalities of SBSs: pruning the search in large human databases, human identification, and human re?identification. With respect to human identification we shed some light on the statistical properties of pertinent parameters related to SBSs, such as employed traits and trait?instances, total categories, size of authentication groups, spread of effective categories and correlation between traits. Further we introduce and elaborate on the event of interference, i.e., the event where a subject picked for identification is indistinguishable from another subject in the same authentication group. Focusing on search pruning, we study the use of soft biometric traits in pre-filtering large human image databases, i.e., in pruning a search using soft biometric traits. Motivated by practical scenarios such as time?constrained human identification in biometric-based video surveillance systems, we analyze the stochastic behavior of search pruning, over large and unstructured data sets which are furthermore random and varying, and where in addition, pruning itself is not fully reliable but is instead prone to errors. In this stochastic setting we explore the natural tradeoff that appears between pruning gain and reliability, and proceed to first provide average?case analysis of the problem and then to study the atypical gain-reliability behavior, giving insight on how often pruning might fail to substantially reduce the search space. Moreover we consider actual soft biometric systems (nine of them) and the corresponding categorization algorithms, and provide a number of experiments that reveal the behavior of such systems. Together, analysis and experimental results, offer a way to quantify, differentiate and compare the presented SBSs and offer insights on design aspects for improvement of such systems. With respect to human re?identification we address the problem of pose variability in surveillance videos. Despite recent advances, face-recognition algorithms are still challenged when applied to the setting of video surveillance systems which inherently introduce variations in the pose of subjects. We seek to provide a recognition algorithm that is specifically suited to a frontal-to-side re-identification setting. Deviating from classical biometric approaches, the proposed method considers color- and texture- based soft biometric traits, specifically those taken from patches of hair, skin and clothes. The proposed method and the suitability of these patch-based traits are then validated both analytically and empirically. Deviating from security related themes, we focus on a completely different application: employing soft biometrics in evaluation of female facial aesthetics. This approach is novel in that, in the context of female facial aesthetics, it combines soft biometrics with previous approaches on photo quality and beauty assessment. This study helps us to understand the role of this specific set of features in affecting the way humans perceive facial images. Based on the above objective parameters, we further construct a simple linear metric that suggests modifiable parameters for aesthetics enhancement, as well as tunes systems that would seek to predict the way humans perceive facial aesthetics. Moreover using the designed metric we evaluate beauty indices with respect to aging, facial surgery and females famous for their beauty. We simulate an automatic tool for beauty prediction with both realistic accuracy and performance. Remaining in the realm of human perception, we also provide a comparative study of different access control systems based on fingerprint, PIN, soft biometrics and face recognition. Towards comparing these systems, we design real?life access control interfaces, each based on the above mentioned methods, and then proceeded to empirically evaluate the degree of usability for each of these interfaces. Our study is based on the recorded assessments of a set of users who rated their interaction with each interface, in terms of privacy, ease of use, user-friendliness, comfort and interaction time. The results reinforce, from a usability point of view, the employment of novel biometric authentication methods as viable alternatives to the currently predominant PIN based methods for access control. Overall this dissertation has contributed the following: ? identification and introduction of novel applications for soft biometrics, such as human identification (bag of soft biometrics), re?identification as well as aesthetics prediction ? development of theoretical framework for SBSs in the applications: pruning the search and human identification ? application of the developed theoretical framework on existing SBSs ? construction of a novel image processing tool for classification of soft biometric traits and employing such a tool in challenging scenarios ? obtaining evidence for the high user friendliness of soft biometric based control access systems.