Color Face Recognition for low-resolution face images


2007 . 06 ~ current




2.     Introduction


In many current face recognition applications, such as video surveillance security and content annotation in web environment, low-resolution faces are commonly encountered and negatively impact on reliable recognition performance. Especially, the recognition accuracy of current intensity-based FR systems can be dropped off significantly if the resolution of facial images is smaller than a certain level (e.g., less than 20 x 20 pixels). To cope with low-resolution faces, this research proposes a novel color FR method, which is not only simple in implementation, but also guarantees extended applicability to most of the FR applications. Experimental results show that color features decrease recognition error rate by at least an order of magnitude over intensity-driven features when low-resolution faces (25 x 25 pixels or less) are applied to three FR methods.


4.      The proposed color face recognition framework


We formulate the base-line color FR framework that can make efficient use of facial color features to overcome low-resolution faces. RGB color face images are first converted into another different color space (e.g, YCbCr color space). Let  be a color face image generated in the color space conversion process. Then, let  be a  spectral component vector of  (in the form of column vector by lexicographic ordering of pixel elements of two-dimensional spectral images), where  and  denotes dimensional real space. Then, face vector is defined as augmentation (or combination) of each spectral component  such that  where   and  represents the transpose operator of matrix. Note that each  should be normalized to zero mean and unit variance prior to their augmentation. Face vector  can be generalized in that, for  the face vector could be defined by gray-scale only while, for  it could be defined by spectral component configuration like YCbCr or YQCr by column order from YCbCr and YIQ color spaces.

Most subspace FR methods are separately divided into the training and testing stages. Given a set  of  color face images,  should be first rescaled into the prototype template size to be used for creation of a corresponding face vector With a formed training set  of  face vector samples, feature subspace is trained and constructed. Rationale behind the feature subspace construction is to find a projection matrix  by optimizing criteria to get lower dimensional feature representation  where each column vector  is a basis vector spanning the feature subspace,  and  It should be noted that  For the testing phase, let  be a gallery (or target) set consisting of  prototype enrolled face vectors of known individuals, where  In addition, let  be an unknown face vector to be identified or verified, denoted as probe (or query), where  To perform FR tasks on probe,  and are projected onto the feature subspace to get corresponding feature representations such that



where   and  A nearest neighbor classifier is then applied to determine identity of  by finding the smallest distance between  and  in the feature subspace as following:



where  returns a class label of face vector and  denotes distance metric. In order to exploit why role of color is getting significant as face resolution is decreased within our base-line color FR framework, theoretical analysis will be given in the following sections.


5.      Experiments


In the practical FR systems, there are two possible FR approaches to perform FR tasks over lower resolution probe images. The first method is to prepare multiple training sets of multi-resolution facial images, and then construct multiple feature subspaces, each of which is charged with a particular face resolution of probe. Alternative method is that lower resolution probe is reconstructed to be matched with prototype resolution of training and gallery facial images by adopting resolution enhancement or interpolation techniques. The second method would be appropriate in typical surveillance FR applications in which high-quality training and gallery images are usually employed, but probe images transmitted from surveillance cameras (e.g., CCTV) are often low-resolution. To demonstrate the effect of color on low-resolution faces in both FR scenarios, two sets of experiments have been carried out in our experimentation, namely Experiment 1 and 2. The Experiment 1 is to assess the impact of color on recognition performance with varying face resolution of probe given multi-resolution trained feature subspaces. On the other hand, the Experiment 2 is to conduct the same assessment when a single-resolution feature subspace trained with high-resolution facial images is only available to the actual testing operation.

Fig. 1 shows examples of facial images containing face resolution variations used in our experiments.

Fig. 1. The examples of facial images from color FERET according to six different face resolutions. Low-resolution observation below original 112 x 112 pixels is interpolated using nearest neighbor interpolation.


5.1  Experiment 1


Fig. 2 shows CMC curves for the identification rate (or BstCRR) comparisons between the gray-scale and color features with respect to six different face resolutions in PCA, FLDA, and Bayesian FR methods. As can be seen in CMC curves obtained from gray-scale R feature (in the left side of Fig. 5), the differences in BstCRR between face resolutions of 112 x 112, 86 x 86, and 44 x 44 pixels are relatively marginal in all three FR methods. However, BstCRRs obtained from low-resolution of 25 x 25 pixels and below are tended to be significantly deteriorated in all three FR methods. For example, for PCA, FLDA, and Bayesian methods, rank one BstCRRs (identification rate of top response being correct) decline from 77.20%, 83.69%, and 82.46% to 56.03%, 37.29%, and 62.32%, respectively, as face resolution is reduced from 112 x 112 to 15 x 15 pixels.






Fig. 2. Identification rate (or BstCRR) comparison between gray-scale and color features with respect to six different face resolutions of each pair of training, gallery, and probe facial images in three FR methods. The graphs on the left side were resulted from gray-scale feature R, while those on the right side were generated from color feature RQCr for each face resolution. (a) PCA. (b) FLDA. (c) Bayesian.


5.2  Experiment 2


In Experiment 2, face resolution of training images was fixed as 112 x 112 pixels, while the resolution of probe was varied as six different resolutions depicted in Fig. 4. Since the high-quality gallery images are usually pre-registered in FR systems before testing probes, we assume that the resolution of gallery is the same as the training facial images, i.e., 112 x 112 pixels. In Experiment 2, R from RGB color space was used as gray-scale feature. Due to the best performance from Experiment 1, RQCr was adopted as a color feature.

Fig. 3 shows CMC curves with respect to six different probe resolutions in the both cases of gray-scale (in the left side) and color features (in the right side) in PCA, FLDA, and Bayesian. To obtain a low-dimensional feature representation for a lower face resolution probe, the probe has been up-sampled to have the same resolution of training faces by using a cubic interpolation technique in Fig. 3. From Fig. 3, in case of a gray-scale feature, we can see considerable identification rate degradation in all three FR methods, considering low-resolution (25 x 25 pixels and below) probes compared to relatively high-resolution counterparts (above 44 x 44 pixels). Especially, similar to the results from Experiment 1, identification rate resulting from FLDA is significantly deteriorated at low-resolution probes. The margins of rank one identification rate between 112 x 112 and each 25 x 25, 20 x 20, and 15 x 15 pixels gray-scale probes in FLDA are 25.66%, 43.77%, and 62.41%, respectively. In case of color feature, BstCRR improvement is made at all probe face resolutions in all three FR algorithms. As expected, face color information greatly improves identification performance obtained from low-resolution probes (25 x 25 pixels and below), compared to gray-scale feature. In PCA, by incorporating color feature, the BstCRR margins between grayscale probe of the 112 x 112 resolution and color probe of the 25 x 25, 20 x 20, and 15 x 15 resolutions are reduced to 3.33%, 4.77%, and 8.02%, respectively. In FLDA, these differences are decreased to 6.65%, 7.28%, and 11.60% at 25 x 25, 20 x 20, and 15 x 15 resolutions, respectively. In addition, in Bayesian, 1.47%, 2.61%, and 5.64% performance margins decrease are achieved with the above three different probe resolutions thanks to color feature.






Fig.3. Identification rate comparison between gray-scale and color features with respect to six different face resolutions of probe images. The graphs on the left side were resulted from R as a gray-scale feature from RGB color space, while those on the right side were generated from RQCr as a color feature for each face resolution. Note that a single feature subspace trained with face images having resolution of 112 x 112 pixels was given to test probe images with varying face resolutions. (a) PCA. (b) FLDA. (c) Bayesian.


6.     Discussion

Traditionally, low-resolution FR modules have been extensively used in video surveillance-like application. Recently, FR applications in web environment are getting increasing attention due to the popularity of online social networks (e.g., Myspace and Facebook) and their high commercialization potentials [4-7]. Under web-based FR paradigm, many devices such as cellular phone cameras and web cameras often produce low-resolution or low quality face images which, however, can be used for recognition purposes [4-5]. As shown in our experimentation, color-based FR outperforms gray-scale based FR over all face resolutions. Particularly, thanks to color information, both identification and verification rates obtained using low-resolution 25 x 25 or 20 x 20 templates are comparable to rates obtained using much larger gray-scale images such as 86 x 86 pixels. Moreover, as shown in Fig. 1, the face DB, used in our experimentation, contain images obtained under varying illumination conditions. Hence, the robustness of color in low-resolution FR appears to be stable with respect to the variation in illumination, at least, in our experimentation. These results demonstrate that facial color can be reliably and effectively utilized in real-world FR systems of practical interest, such as video surveillance and promising web applications, which frequently have to handle the low-resolution face images taken under uncontrolled illumination conditions.



 * Contact Person: Prof. Yong Man Ro (


l  1.       Jae Young Choi, Yong Man Ro, and Konstantinos N. Plataniotis, Boosting Color Feature Selection for Color Face Recognition, IEEE Trans. on Image Processing, vol. 20. no. 5, pp. 412-430, Feb., 2011.

l  2.       Jae Young Choi, Yong Man Ro, and Konstantinos N. Plataniotis, A Comparative Study of Preprocessing Mismatch Effects in Color Image based Face Recognition, Pattern Recognition, vol. 44. no. 40, pp. 412-430, Feb., 2011.

l  3.       Jae Young Choi, Yong Man Ro, and Konstantinos N. Plataniotis, Color Face Recognition for Degraded Face Images, IEEE Trans. on Systems. Man and Cybernetics- Part B, vol. 39. no. 5, Oct., 2009.


l    Jae Young Choi, Seungji Yang, Yong Man Ro, and Konstantinos N. Plataniotis, Color effect on the face recognition with spatial resolution constraints, IEEE Intl Conf. on International Sysmposium on Multimedia (ISM), Berkeley, California, USA, 2008.


    5.    Jae Young Choi and Yong Man Ro, Color Correction using Color Flow Eigenspace Model in Color Face Recognition,             SPIE Intl Conf. on Electronic Imaging, San Jose, USA, 2008.


    6.    Seung Ho Lee, Jae Young Choi, Yong Man Ro, and Konstantinos N. Plataniotis, Color Component Feature Selection In             Feature Selection In Feature-Level Fusion Based Color Face Recognition, " IEEE Intl Conf. on  Computational             Intelligence (WCCI), Spain, 2010. (accepted)