Benchmarking Video Face Super-Resolution Algorithms
Nasser Nasrabadi, Jeremy Dawson and Moktari Mostof
In our previous CITeR project entitled “Face Super-Resolution From a Single Image,” we demonstrated that single face images of size 16×16 can be super-resolved to improve the performance of Face Recognition (FR) algorithms. However, performing FR from very small size face images (8×8) contained in low quality surveillance video camera frames remains a challenging task. Previous research has demonstrated that using video super-resolution algorithms (making use of sub-pixel movements between adjacent frames) can further improve the quality of the super-resolved images when compared to a single input image. Low-resolution surveillance videos captured under non-ideal conditions are often contaminated by blur, non-uniform lighting, variations in facial expression, and non-frontal face pose, which will adversely affect the performance of Super-Resolution (SR) algorithms. Recently, many Video Super-Resolution (VSR) algorithms based on Deep Learning (DL) technology have been developed for low-quality videos in surveillance applications. Most of the DL-based VSR approaches need to align multiple frames (i.e., estimate interframe motion) to a reference frame before generating a High-Resolution frame by fusing the motion-compensated Low-Resolution frames. However, effective and efficient utilization of temporal information remains a challenging task since complex motion is difficult to model and can introduce adverse effects if not handled properly. For face super-resolution algorithms optimized for a FR task, prior facial knowledge (i.e., facial landmark heatmaps and parsing maps) can also be leveraged to better super-resolve face images, which have been incorporated in a number of face related VSR algorithms. In this project, we propose to conduct a comprehensive evaluation of existing DL-based VSR models, and analyze their performance on FR for faces at a distance. The outcomes of this study are two-fold: 1) For end users of DL-based VSR, our benchmark results can serve as a guide for selecting the appropriate VSR algorithm for a given video surveillance quality. 2) For algorithm developers, our in-depth analysis of VSR algorithms will uncover possible future directions to further optimize VSR algorithms for higher FR performance. Another research task is to quantify the effect of the prior facial knowledge on facial VSR algorithms as well as the image quality of the aligned face frames.
Biometric and Biographic Data Cleanup with Deduplication and Quality Scoring
Guodong Guo (WVU), and Xin Li (WVU)
In biometrics research & development, data is more and more important, including both the image data and biographic data. As the data set gets larger and larger, there will be a higher probability to involve various issues, such as the identity label noise, incorrect biographic annotation, and duplication of images or identity records. Furthermore, there will be images with different qualities as well in wild data collection. We call these biometric data issues (BDI), which exist not only in commercial biometric product development, but also in the datasets with many government agencies, e.g., FBI, DHS, DoD, IARPA, etc. Therefore it is very crucial to develop an automated method to address the biometric data issues for both academic research and practical applications. In this project, we propose to study an important problem, i.e., how to develop a computational method to perform biometric and biographic data cleanup in large scale datasets. For this specific study, we focus on face images, and related biographic data, but the developed technique might be applied to other modalities. The developed method can also detect the duplications of identity records, and provide face image quality scoring or assessment. The objective is to explore an efficient method that can cleanse the biometric and biographic data with deduplications, and also perform quality scoring for the image samples in a large scale face dataset. The outcome is an automated tool for biometric data cleansing and a cleansed dataset, which can be delivered to the affiliates.
Biometric Data Classification for Large-Scale Database Error Detection and Correction – SP
Jeremy Dawson (WVU), Nasser Nasrabadi (WVU), John Dando (WVU, student)
Operational biometric databases are compiled via aggregation of data collected by numerous personnel
located in many disparate operations. This can lead to manual entry errors in the dataset that significantly impact the ability to match new encounters against an existing watchlist. These errors can consist of modalities entered into the incorrect data field (e.g. a fac image entered in a fingerprint field), mislabeled data (e.g. flat vs. rolled fingerprints), or image entered in an incorrect orientation. The objective of this project is to create lightweight, efficient classifiers that can scour large datasets and flag the errors mentioned above for manual correction. Software containerization and implementation will be performed by the sponsor. Error identification and classification performance will be compared
against GOTS tools designed for similar tasks.
Comparative Detection of Facial Image Manipulation Techniques
Chen Liu / Zander Blasingame
Facial image manipulation techniques are becoming increasingly widespread for both benign and malevolent reasons. To verify whether an image has been altered by such techniques, we propose a comparative technique for detecting which type of a filter has been used. First, we propose to use an autoencoding model to learn the “inverse” filter by treating the filter as the encoder and training the decoder component to learn the mapping from the altered image to the unaltered image. Then using the trained autoencoder pairs, the inverse filter is applied to generate an inverse filtered image. We then propose to use a deep convolutional network trained on genuine faces to determine if an image in question or its inverse filtered image is more likely to be a genuine face. If, for any image generated by the inverse filter, it turns out to be more probable to be genuine than the original image, then it is likely that a particular filter was used to create the image in question.
Contactless Fingerprint Recognition using Smartphone Cameras
Anil Jain
We propose an algorithm (including preprocessing and matching) for contactless fingerprint images (i.e. fingerprint images captured from mobile phones). In our preprocessing pipeline, we will enhance the contrast of the fingerprint ridge structure in order to better identify robust and discriminative features (minutiae points) in the fingerprint image. Our matching algorithm will consist of (i) a contactless fingerprint minutiae detector coupled with a contactless minutiae matcher, and (ii) a fixed-length texture representation generator for the contactless images. Our final algorithm will fuse the minutiae representation of the contactless image together with the texture representation of the contactless image. We will evaluate the performance of our method and compare it against two state-of-the-art COTS contactless (mobile phone) fingerprint matching systems on an in-situ dataset collected in Jaipur, India, in March 2018. (309 subjects x2 thumbs x2 index fingers x2 impression) by Crossmatch (touch based) & 2 low-cost commodity smart phone cameras.
Deep Deblurring of Fingerphotos Captured by Smartphones
Jeremy Dawson and Brady Williams
As the mobile e-commerce continues to grow, smart phone biometrics authentication is attracting increased attention for the secure and easy-to-use identification methods. However, ridge quality of fingerphotos captured by a smartphone are often reduced by motion blurring caused by the movement of the smartphone or the user’s finger, and the inability of cameras to properly focus on the fingerprint ridge patterns. This blurring greatly impacts the image quality and subsequent matching efficacy of the
captured print. Only a few of the previous fingerphoto matching techniques have investigated the influence of the image distortions caused by the camera being out of focus or the blurring caused by hand movements during the fingerprint capture. In this project, we propose to develop a deep deblurring algorithm based on the conditional Generative Adversarial Network (cGAN) architecture [1] to defocus and restore motion-blurred fingerphotos captured by a mobile camera. We train a cGAN architecture to map blurred fingerphotos into their corresponding high-quality blur-free fingerphotos. We will also investigate how to incorporate the fingerprint ridge information into our cGAN-based deblurring module. This is one of the key components of our proposed algorithm. We will also extend our cGAN deblurring network to a multi-task cGAN architecture where a blurred fingerphoto is
mapped to it corresponding target ridge patterns and its restored high-quality fingerphoto. By enforcing the reconstruction of target ridge patterns from the blurred fingerphotos, our network is learning to deblur fingerphoto images with sharp ridge patterns that are more recognizable. We will use the Innovatrics commercial fingerprint matcher to evaluate our fingerphoto deblurring algorithm
against a gallery of reference blur-free fingerphotos. Performance criteria for our fingerphoto deblurring will be based on the ROC and DET curves and EER values.
Deepfake Video Fingerprinting – SP
Shan Jia, Yuezun Li, Siwei Lyu and Xin Li
High quality AI-created digital impersonations, known as Deepfakes, have become a serious
problem from 2017, which could irreparably damage public trust in video content. Several publicly available databases have been proposed to promote a variety of detection methods. However, most existing methods evaluate their performance on databases based on single Deepfakes implementation tool. Will different manipulation tools affect the detection performance? We propose to explore this question in this project and develop fingerprinting tools for Deepfake videos. The basic idea is to study how different Deepfake manipulation tools will leave artificial but unique fingerprints in video. This line of research can help improve our understanding toward Deepfake video and optimize the detection performance on heterogeneous datasets.
Deep Spoof Detection for Text-independent Speaker Verification – SP
N. M. Nasrabadi, J. D. Dawson and S. Soleymani (WVU)
Due to several technical advances, notably channel and noise compensation techniques, automatic speaker verification (ASV) systems are being widely adopted in security applications. However, the robustness of these systems against the spoofing attacks is a major concern since even the state-of-the-art ASV systems struggle to sustain against spoofing attacks, such as impersonation, synthetic, converted and replayed speech [1]. A spoof detection framework can consider three feature representations: short-term spectral features, prosodic characteristics, and high-level features which reflect language content and speaker behavior [2]. However, each of these feature representation is susceptible to one or several spoofs [3]. In addition, spoofing countermeasures constructed upon the assumption of a unique spoof, lack the possibility of generalization for unknown attacks. We propose a deep neural network capable of analyzing inputs at different levels of feature representation. This network consists of a deep convolutional network (CNN) and a multi-layer perceptron (MLP) network. The CNN is fed with the short-term spectral features, while the MLP network processes the other two levels of speech features. The speech signal is divided into multiple fixed-duration segments and the short-term features corresponding to each segment, along with the features from the other two levels of representation from the whole speech signal are fed to the network. These two networks are jointly trained using two training paradigms. These paradigms consider the genuine-spoofed paired and unpaired scenarios. The element-wise average of the output space features for speech segments are considered to determine whether or not a speech signal is spoofed. The performance of the proposed framework will be evaluated on publicly available ASV spoof 2015 [4] and ASV spoof 2019 [1] datasets. To the best of our knowledge no one has used such a comprehensive study of different levels of speech representation for spoofed speech detection using deep learning models.
Detecting DeepFakes using Physiological, Physical, and Signal Features – SP
Siwei Lyu, David Doermann and Srirangaraj Setlur
In this work, we aim to develop forensics technology to expose AI-generated fake videos, including DeepFakes. As these deep learning based, automated ways to synthesize and manipulate videos continue to advance, the resulting videos pose new challenges for existing digital forensic tools. These challenges require innovative approaches that are dedicated to these evolving forms of forgery. Our previous work on Media Forensics indicates that these deep learning models and processing pipelines produce DeepFakes that lack basic physiological signals, have inconsistent physical characteristics, and produce signal level artifacts. Our approach will be exploiting this trace evidence.
Detecting DeepFake Videos by Analyzing Both Audio and Visual Cues
Arun Ross and Xiaoming Liu, Michigan State University
DeepFakes are synthetic renditions or manipulations of images, videos, audio or text that can potentially be used to mislead viewers and readers. A number of methods have been proposed in the literature to detect DeepFakes. Most of them rely on training a deep neural network (DNN) using labeled examples of real videos and DeepFakes. However, the generalizability of such approaches to newer types of DeepFakes, generated by increasingly powerful generative adversarial networks, is often limited. Further, the quest for designing a universal DeepFake detector can devolve into a cat-and-mouse game (the “Moving Goalpost” phenomenon). To address these concerns, we propose to advance the state of the art in DeepFake video detection as follows: (a) utilize both audio and visual cues; (b) utilize principles of digital forensics; (c) harness the power of unsupervised clustering to discover categories of DeepFakes; and (d) design explainable models. One additional contribution of this project will be the generation of DeepFake videos where either (or both) audio and visual tracks will be DeepFakes.
Detecting Face Morphing: Dataset Construction and Benchmark Evaluation – SP
Jacob Dameron, Guodong Guo, and Xin Li (WVU)
Recently, there has been a rise in concern about the vulnerability of facial recognition systems to face morphing attacks [1]. These attacks combine the images of two individuals to form an image that can be used to trick facial recognition systems. Our contribution will be in two parts. First, we propose to construct a new dataset of face morphs for testing detection algorithms. The morphs will be generated by applying multiple algorithms, including generative adversarial network (GAN) based and traditional landmark based (LMA) [2] attacks, to image pairs belonging to publicly available datasets. Second, we will conduct a comprehensive evaluation of existing face morphing detection methods against our dataset. We will rank those methods and explore the possibility of combining them by recently developed bilinear pooling techniques [4].
Determining the Uniqueness of Facial Images in Large Datasets – SP
Jeremy Dawson and Nasser Nasrabadi
One of the desirable characteristics of a biometric trait is its uniqueness, a measure of how different that trait is among large groups of individuals. It is this uniqueness that makes automated biometric identification possible. However, as datasets become large, the uniqueness of a given trait decreases as the chance of two individuals with similar traits will appear increases. For this reason, a method of measuring the uniqueness of an individual’s biometric traits in a given dataset is needed. The goal of this work is to design an algorithm that can provide a scalar value (e.g., similar to entropy defining value of an information in an image used in compression technology) determining the uniqueness of a biometric trait which is also related to performance of biometric trait matcher. We propose to use the concept of a common image manifold (subspace) that captures the commonality (shared features) of all the images in our database. Here, we will adapt this concept to develop a measure of the ‘average’ facial features (i.e., a lowdimensional facial subspace) in a dataset. Then, rather than comparing facial features from two or more individuals to perform identification, the facial image of a given individual will be mapped onto the orthogonal complement of this shared facial subspace and the L2 norm of the resulting residual face image is calculated, which represents how much this face deviates from the shared face subspace, which also shows the uniqueness of that persons’ face. This uniqueness feature will then be correlated to match scores from commercial and academic facial recognition algorithms to determine the impact of this uniqueness measure to the capability of the matcher to distinguish between different people. We will investigate facial uniqueness for different age and ethnic groups and relate it to face recognition algorithms. While this project will focus on face images, the same concept could be extended to other image-based biometric traits, such as fingerprint and iris.
Digitally Altered Data: Finding the Original from Near-Duplicate Biometric Images – SP
Arun Ross, Michigan State University
The goal of this project is to automatically identify the original image from a set of digitally altered
biometric images and also determine the relationship between them. The alterations can be photometric (e.g., image filters) and/or geometric (e.g., rotation, zooming). Such modifications can be applied to the original image in a sequence resulting in a set of near-duplicates that may be visually indiscernible from the original. We propose to develop an end-to-end scheme that will accept a set of near-duplicate images as input and produce two outputs: (a) The first will be the original image from the given image set, and (b) the second will be a hierarchical structure indicating the order in which these images were transformed. The order will represent how the images are related to each other, viz., the
sequence in which they were derived starting from the original image. Wherever possible, the scheme will indicate the type of alterations that were applied to the images. The proposed scheme will be applicable to both manually edited images as well as those modified using commercial software, e.g., Photoshop. A web interface will be developed to showcase the methods developed in this work. The project is expected to benefit image forensic applications.
Digitization of 10-Print Card Fingeprints Using Cellphone Cameras – SP
Jeremy Dawson (WVU), Nasser Nasrabadi (WVU)
Defense agency criminal investigators often experience situations in which they have access to a card-based 10-print fingerprint record which cannot be transferred to them or scanned on site due to lack of equipment. Often, the investigator will capture a photo of the fingerprints on the card. However, there are some uncertainties as to whether or not these images can be used for matching due to variations in the imaging process from person to person, as well as variation in 10-print card type. The goal of this project is to evaluate the use of cellphone cameras in capturing photos of inked 10-print cards to see if the resulting images can approach Appendix F quality and resolution standards. WVU has a collection of 1000+ 10-print cards of varying image quality that can be used for this study.
Fingerprint Segmentation for Juveniles and Adults
Keivan Bahmani (CU), Stephanie Schuckers (CU)
Fingerprints have been used for biometric recognition in children for applications such as border crossings, health benefits, food distribution, etc. In many applications, the fingerprint processing pipeline has to initially segment the 4 fingerprint slaps into individual fingerprints. However, most of the current fingerprint segmentation algorithms have been trained only on adult datasets [1-2]. Compared to adults, juvenile fingerprints have different sizes, spatial characteristics and quality [3]. This could potentially lead to suboptimal fingerprint segmentation which subsequently leads to lower identification accuracy. In this work, we aim to develop new fingerprint segmentation models capable of effectively processing both adults and juvenile fingerprints. We investigate the possibility of fine tuning the open source NIST NBIS fingerprint segmentor [4] as well as developing new deep learning based fingerprint segmentation models.
High Resolution Face Completion for Authentication under the COVID-19 Pandemic
Gianfranco Doretto, Don Adjeroh (WVU)
The global COVID-19 pandemic has changed everything – from how we work to how we interact with others. The use of face coverings of different types is one approach that has been used to curb the spread of the virus and to reduce infection. From the human identification perspective, these face coverings pose a significant challenge. Apart from the obvious problem of severe occlusion, the type and extent of the face covering has a significant impact on automated face authentication. This is further complicated by the additional issues of pose, facial expression, illumination and background lighting, which are traditional problems in face recognition. Though the occlusion problem has been studied in the past, and the above problems have been investigated independently, there has been little work on a comprehensive investigation on the combined impact of these factors on face recognition performance, and this has motivated the ongoing face recognition with masks vendor test of NIST [1]. In this project, we propose to build on our recent work on ALAE (adversarial latent auto encoders) [7] to develop the first high resolution face completion approach for “uncovering” face images. We will collect and create datasets for training, testing, and adapting deep learning models for the approach. We will then thoroughly evaluate the approach to unveil its potential for improving face recognition based on images of faces with different coverings.
Homomorphically Encrypted Matching and Search – SP
Anil Jain
We propose methods to do both 1:1 biometric matching (authentication) and 1:N biometric matching (search) in the encrypted domain, using fully homomorphic encryption. Since matching is done within the encrypted domain, without a need to decrypt, template security is significantly enhanced. Furthermore, homomorphic encryption does not negatively impact the recognition, i.e., biometric matching accuracy. Existing methods are unable to easily leverage homomorphic encryption due to its incredible computational complexity. Our specific contributions will be: fast encrypted matching, using a combination of cryptographic (i.e. encoding schemes) and machine learning techniques (i.e. dimensionality reduction). We plan to demonstrate the practicality of our system on face, fingerprint, and iris image datasets at scale.
Investigating cardiac waveform and related physiological parameters in personal identification
Jun Xia
Recently, there has been increasing interest in utilizing the electrocardiogram (ECG) waveforms as
a biometric marker for authentication and identification purposes. ECG signal represents the electrical activity of the heart, and it possesses high uniqueness among individuals. The origin of the signal also enables liveness detection. However, ECG waveform only represents a portion of the cardiac and circulation system. The heart and blood circulation are also characterized by other parameters, such as the pre-ejection period and the pulse arrival time. These cardiac parameters can be detected by other cardiac monitoring techniques, such as impedance cardiography. It is unclear whether other cardiac waveforms or parameters are superior to ECG for biometric applications. This proposal aims to answer this question through a comprehensive analysis of different cardiac parameters.
Joint Face Pose Estimation and Frontalization
Nasser Nasrabadi and Jeremy Dawson
The performance of current face recognition (FR) algorithms degrades significantly when comparing frontal to extreme pose face images under unconstrained conditions [5]. In our previous CITeR project “Deep Profile-to-Frontal Face Verification”, CITeR project #19S-03W, we developed a Couple Generative Adversarial Network (CpGAN) to directly match profile to frontal faces in a latent embedded feature space [6]-[7]. However, in order to use off-the-shelf commercially available face matchers, we need to
develop a pre-processing module that can directly synthesis photorealistic frontal views from face photos at extreme poses (profiles). We hypothesize that there is an inherent nonlinear mapping (frontalization) between profile to frontal (or frontal to profile) faces, and consequently, their discrepancy can be bridged by a domain adaptation mapping based on a conditional GAN (cGAN) architecture [1]. In this project, we propose a multi-task cGAN-based face frontalization algorithm to synthesize a canonical frontal-view from any extreme pose. Our algorithm also uses a multi-task framework to simultaneously rotate the face to a frontal-view and give an estimate of the pose of the input probe face. One advantage of our method is that it does not rely on any 3D knowledge about the face geometry or shape, the frontalization is performed through sheer data-driven learning. To generate photorealistic frontal faces, we incorporated several key constraints such as the self-similarity of face across left and right halves, preserving identity while synthesizing the frontal view, the perceptual loss and reconstruction loss between the synthesized and frontal-view faces. We will base our quantifiable assessment on the following criteria: 1) the overall increase in face recognition accuracy using our profile-to-frontal face frontalization algorithm, 2) the effect on face recognition accuracy using our face frontalization algorithm for different pose angles, 3) a comparison of the face recognition accuracy of frontalized profile-to-frontal to that of frontal-to-frontal on different databases, 4) we will plot the ROC and DET curves and EER values.
Leveraging finger relationships for 2-finger authentication on mobile devices
Sergey Tulyakov
The new generation of mobile fingerprint scanners, such as Qualcomm’s 3D Sonic Max, allows
simultaneous capture of 2-finger prints. Although it is possible to use the traditional techniques of slap image segmentation and recognition of individual fingers for such sensors, such approaches could be suboptimal as they do not take advantage of the relative positions of fingers. We hypothesize the existence of following relationships between individual finger prints: positioning of prints due to lengths of fingers, similar angle of in-plane rotation (yaw) due to hand orientation during scan, similar shifts in captured skin regions due to hand axis rotation (roll) and out-of plane rotation (pitch). Because of great variability of hand placement on mobile devices, we expect such relationships to be an important
factor for mobile multiple finger recognition methods. We propose the collection of the 2-finger image database emphasizing such variability and allowing better benchmarking of algorithms for multiple finger images. A baseline algorithm for extracting finger relationship features and their utilization for recognition will be developed as well.
LivDet 2021: Liveness detection competition for fingerprint, iris, and face
S. Purnapatra (CU), D. Yambay (CU), S. Schuckers (CU), T. Bourlai (WVU)
Biometric recognition systems are vulnerable to artificial presentation attacks, such as molds made of silicone or gelatin for fingerprint and patterned contacts for iris. Liveness Detection, or Presentation Attack Detection (PAD), has been a growing field to combat these types of attacks. LivDet 2017 showcased 9 total competitors for the fingerprint system competition and 3 competitors for iris algorithms. We propose to host LivDet 2021 with a similar public dataset for training algorithms and systems. In addition, we propose to build on the cross-dataset competition and the matching with liveness which began in 2017, which would be shared with researchers anonymously through BEAT platform to support advancement in PAD research. Additionally, iris LiveDet competition will also be hosted for PAD. As a possible addition (pending budgetary considerations), we would like to add LivDet-Face including both visible and NIR images of spoof and live faces.
Multimodal Feature Fusion for Automobile Abnormality Detection
Srirangaraj Setlur, Dennis Fedorishin, Deen Mohan, Nishant Sankaran
After successfully employing deep learning and audio processing methods to detect vehicle engine
anomalies through engine audio recordings, the groundwork for developing an “automobile fingerprint” has been created through our gateway project. However, ACV Auction’s condition report dataset has other modes of information that may be useful for further extracting features of automobile anomalies. These include accelerometer-recorded vibration data, expert text descriptions of vehicle issues present, and active engine OBD II diagnostic codes. Through this continuation project, we seek to develop a fusion framework to further increase the performance of detecting and identifying anomalies
of vehicle engines. Methods to filter out wind, voices and other ambient noise will also be explored to improve performance. In addition, we will investigate the use of vehicle undercarriage images to detect anomalies such as excessive rust and frame damage by using ACV Auction’s dataset of Virtual LiftTM images if time permits.
On the uniqueness of facial identity – SP
Xiaoming Liu, Anil K. Jain, Michigan State University
Given a face image, the uniqueness of facial identity is defined as the average feature space distance between its identity latent vector to the vectors of its top K most similar subjects, divided by the averaged distance of all faces in an image set. The lower the uniqueness value, the more likely it will be misclassified as other identities. The objectives of this work include 1) quantify the uniqueness of faces in face benchmarks, especially on the variation of uniqueness and factors that are correlated with the uniqueness; 2) design models to estimate uniqueness of a face image; 3) design face recognizer that can generate latent vectors where the minimal uniqueness within an image set increases; 4) exploit the possibility of applying the uniqueness to anti-deepfake, assuming that the synthetic faces might have low probability in the identity space (i.e., higher uniqueness). Essentially, we would like to address: (i) how are the cotton balls placed in the space? (ii) what determines their placements? and (iii) can they be more evenly placed to minimize misclassification?
Preserving the Privacy of Face Embeddings in Modern Face Recognition Systems
Vedrana Krivokuća Hahn and Sébastien Marcel (Idiap – CH)
Modern face recognition systems, based on deep learning, convert face images into highly
representative features called “embeddings”. The possibility of ‘inverting’ an embedding to recover the original face image is already being explored (with promising results), which represents a threat to the privacy of face recognition system users and the security of the systems themselves. So, the aim of this project is to investigate effective strategies for mitigating these threats by converting face embeddings into a non-invertible, renewable representation, thereby protecting the originals. We plan to start with a method that we initially designed for securing i-vectors in speaker recognition systems, PolyProtect, which is based on multivariate polynomials applied to real-number vectors. Our focus will be verification (1-to-1) systems only.
Privacy Enhancing Biometric Sensors – SP
Arun Ross, Michigan State University
The goal of this project is to outfit biometric sensors (e.g., face cameras) with privacy enhancing features. In particular, we will develop a scheme where the acquired images are automatically perturbed at the sensor level such that only specific information can be extracted from the ensuing images while other types of information are obfuscated. For example, a facial image may be perturbed such that attributes such as age, sex and ancestral original cannot be extracted by automated attribute classifiers, but automated biometric matching is retained (soft-biometric privacy). Conversely, the image may be perturbed such that it cannot be used for biometric matching, but attributes such as age, expression or race are retained. Since the aforementioned perturbations are performed at the sensor level, the original image is never made available. When such cameras are deployed in operational scenarios, certain privacy guarantees can be empirically guaranteed. This will result in user-controllable privacy.
Privacy-Preserving Biometric-Based Authentication using Secure Multi-Party Computation – SP
Marina Blanton
This project focuses on improving efficiency of privacy-preserving biometric-based authentication techniques, where users are able to authenticate to remote services using their biometric data, but without having their biometry stored in the clear on remote servers. The users will rely on mobile devices such as smartphones to read a biometric sample and facilitate privacy-preserving interaction to perform biometric matching without disclosing biometric data to the remote server. Instead of using homomorphic encryption which incurs significant overhead, this project employs alternative, less known outside of cryptographic community techniques such as secret sharing and garbled circuit evaluation which offer superior performance. We will further optimize previously developed secure biometric matching protocols for this application to improve their speed and scalability for popular biometric modalities (e.g., face, voice, fingerprints, and iris).
Revisiting Voice Biometric Entropy with Demographic Impacts – SP
Wenyao Xu, Srirangaraj (Ranga) Setlur
Recent efforts in voice biometrics are primarily focused on elevating the accuracy during user authentication and making the technology convenient to use in real-world application. The notion of “voice is unique” is regarded as a known fact, and its degree of uniqueness is often related to the evaluation metrics such as false positive and negatives. In this work, we aim to revisit the entropy of human voice using qualitative and quantitative efforts to answer a fundamental, yet never explored question, “how much unique is your voice?”, with demographic impacts. A set of statistical and technical efforts include: (1) Exploring the biological contributors to variability in human voice; (2) Modeling and analyzing the statistical relation between voice’s entropy with time and domain-specific features (e.g., frequency domain) and matching algorithms; (3) Evaluate the entropy across speech and non-speech from a large set of participants with different demographics and health histories. If successful, this study will aid in providing a deeper understanding of voice biometric uniqueness and may even disclose a generalized wisdom to hardness the vulnerability of voice biometrics.
Toolkit for Explainable AI in Biometric Recognition
Stephanie Schuckers (CU), Mahesh Banavar (CU), Keivan Bahmani (CU)
Deep Neural Networks (DNNs) demonstrated remarkable success in many aspects of biometrics. However, due to their complexity and multiple layers of abstractions, it is not easy to obtain a clear, interpretable relationship between the inputs and outputs of a DNN. The field of XAI (eXplainable AI) aims to address this issue by explaining and representing this relationship in a human understandable terms. This explainable relationship plays a crucial role in various use cases such as decision making involving both humans and DNNs, verifying and debugging generalization of the model, efficient
retraining of models and improving transparency to prevent unexpected behavior and unintended discrimination [1]. Here, we aim to provide a unified framework and toolkit for XAI algorithms specific to biometric recognition. The proposed framework will incorporate the necessary adjustments to the biometric data, including aligning the face, marking landmarks, and applying pre-processing functions in case of fingerprints. At the completion of the project, the toolkit will be ready to be deployed by affiliates through an easy to use Graphical User Interface (GUI) and be capable of receiving and converting models between various deep learning platforms. Affiliates will be provided both high and granular level code access.
Touchless fingerprint biometrics at border crossing with user mobile phones
Nalini Ratha
In the current COVID19 pandemic situation and the post COVID19 world, touchless biometric
modalities would be of significant interest to the community. Finger-selfies on user owned phones can provide a good alternative to traditional touch-based fingerprint recognition at border crossing. The usage of finger-selfies for authentication can be two-fold: (i) both the query and probe samples are finger-selfie images and (ii) in the scenarios of remote authentication, finger-selfie may be matched with livescan fingerprints present in the legacy databases. Existing studies primarily focus on designing algorithms in controlled environments with limited devices. However, matching touchless fingerprints between various mobile devices in unconstrained environments provide very poor results. This research aims to study the impact of different mobile phones on accuracy, effect of user collected fingerprint vs operator collected data, impact of a video vs single shot image and design an efficient algorithm for automated finger-selfie recognition.
Uniqueness and permanence of iris – SP
Priyanka Das (CU), Stephanie Schuckers (CU), Joseph Scufka (CU), Natalia Schmid (WVU), Matt Valenti (WVU)
Uniqueness and permanence are two of the inherent characteristics of biometrics. The random variation and complexity in the iris pattern is the basis for uniqueness in iris. Research in the past decade explored techniques to identify iris structures that are potentially noiseless and stable. Prior work has showed that the performance of iris recognition improves when considering the “stable bits” for matching. We studied effect of time difference between enrollment and probe on iris recognition performance in children. We noted a slight decay in median similarity score with time, though it does not impact the tails of the distribution of iris recognition performance over a period of 3 years. Uniqueness of the iris features has been well established in literature. However, the capacity of the iris recognition system (end to end from presenting the iris to the sensor to decision making) i.e. the maximal number of users at which the iris system reaches a particular FAR, is impacted by multiple factors – presentation of the iris to the system by the user (head tilt with respect to the sensor, angle of view, distance from the sensor), sensor based noise, quality factors of the captured image (dilation, obstruction in the iris due to eyelash, hair etc., area of the iris available for analysis), matching algorithm (how it deals with rotation, delta dilation common iris area between mated pair of images, features used, pre-processing steps) and the decision making process. We propose to investigate the longitudinal stability of iris features and the uniqueness of an iris recognition system in presence of variability factors like aging and quality factors. We propose to model an iris recognition system and statistically estimate the capacity of the system in presence of the variabilities. We also propose to investigate the longitudinal stability of the “stable bits” to investigate the root cause of decay in similarity score. This project will address the following: Model an end to end iris recognition system to estimate the system’s capacity in light of various variability factors (aging and quality) and their impact on False Match Rate/ False Non-Match Rate. How stable are the iris stable bits longitudinally?
Video Series Expansion–Biometric Recognition Technology Under Scrutiny: Public Outreach on Technology Fundamentals – SP
S Schuckers, L. Holsopple, D Hou, M. Banavar (CU)
Biometric recognition technology has had a tremendous explosion in recent years. However, with that broad reach, has come increased scrutiny of the technology and public discussion around its uses, as well as creation of new regulations and policies providing a framework and limits. Underlying this discussion is confusion around how the technology works and how it differs for various application use cases. Misunderstandings around the technology may lead to poor regulations or, in certain cases, bans on the technology. Currently, we are creating three videos in a series that focus on the fundamentals of biometrics technology (how it functions, different modes of operation, and applications). However, there are many more questions which the decision makers and broader public have in terms of where the data is stored, who has access to it and for what purpose, how vulnerable are systems, whether systems are biased. The purpose of this project is to add three short educational videos to focus on these harder questions. The videos will be broadly disseminated, but the intended audience is policy makers, media, and technology decision makers. These videos will NOT provide recommendations for specific policies and laws, but rather provide foundational material on which good policies can be made.
Wavelet-Based Morphed Artifacts Detection – SP
Nasser M. Nasrabadi and Jeremy Dawson (WVU)
In our current differential morphed face detection algorithm (Detecting Differential Morphed Faces Using Deep Siamese Network), we have demonstrated a high detection performance. However, given a single image, detecting if it is a morphed face photo is still a challenging problem. To detect and defend against such morphing attacks, we need to identify the image artifacts that arise due to the morphing process. From our previous research on morphed faces [7] and adversarial biometric templates [8], we have demonstrated that artifacts due to morphing reside in the high frequency regions of the morphed images. Therefore, in this project, we are proposing to use a wavelet transform to decompose a morphed image into its sub-band components, and identify the high-frequency sub-bands that distinguish the morphed faces from natural face photos. The first goal is to identify the sub-bands where the morphing artifact would appear and then design a classifier using these subbands. Thus, we will develop an optimization algorithm to find the optimal wavelet decomposition of the training images (morphed and regular face images) to identify the sub-bands that will provide the maximum discrimination between morphed and regular face photos. Using these identified sub-bands where the morphed artifacts could reside, we will train a binary classifier (i.e., SVM, MLP) to differentiate between the morphed and regular face photos.