: : : D:\Ȩ \Ʈ\123.files\image002.jpg: : : http://ivylab.kaist.ac.kr/image/demo/project_title.gif

: : : D:\Ȩ \Ʈ\LabHomepage_3DResearch\AttentionModelbasedVCA_image.files\image004.gif

: : : D:\Ȩ \Ʈ\LabHomepage_3DResearch\AttentionModelbasedVCA_image.files\image005.gif

Collaborative Face Recognition for Face Annotation Applications in Personal Photos Shared on Online Social Networks

2010. Oct ~ Present  


: : : D:\Ȩ \Ʈ\LabHomepage_3DResearch\AttentionModelbasedVCA_image.files\image007.gif

1. Introduction

l  Online Social Networks (OSNs)

o    The number of personal photos in OSNs keeps significantly growing due to

  Easy-to-use multimedia devices and online services

  Cheap storage and bandwidth

  An increasing number of people going online

o    Sharing of personal photos in OSNs

  Associated with their photographer

  Being broadcasted out to the online contacts of the photographer

  Joining a collection of billions of other photos

o    Problem: Digital information overload (requirement to automatically organize photos in OSNs

Fig.1. Image Sharing in OSNs

l  Face annotation or name tagging

o    Allows person-based organization and retrieval of photos

o    Considerable practical interest due to high commercial potential

o    Automatic face detection and face recognition (FR)

Fig.2. Face Detection from Personal Photos and Process of Face annotation and Retrieval

l  Architecture of OSNs

o    Centralized

  Centralized server with all user data (e.g., contact list, photos)

  Personal several problems (information silos, ownership of data, and privacy issues)

o    Decentralized

  Attracting more and more interest

  Present several advantages

         Each user has a personal server containing his/her user data

         Giving users more control over data ownership and privacy

Fig.3. Architecture of OSNs

l  Research Motivation:

o    Limitation of traditional FR:

  Appearance-based FR solutions still come with a low accuracy

  Restricted to be developed using centralized FR framework relying on a single FR engine

o    Decentralized OSNs

  Each user will have a personalized FR engines, optimized for recognizing a small set of individuals in question

  Sharing FR engines between a user of the OSN and his/her contacts

o    Development of a novel collaborative FR framework

  Social context in personal photos

  Social network context in OSNs

  Use of multiple FR engines

l  Research Goal: Given a photographer, the goal is to use FR engines of other users for face annotation in personal photos of the given user

Fig.4. Proposed framework of collaborative FR

2. Collaborative Face Recognition

l  Research challenges:

o    How to select expert FR engines?

o    How to merge multiple FR scores in a single decision?

o    How to deal with heterogeneous FR engines? (FR engines using different FR techniques)

Fig.5. Proposed collaborative FR framework in an OSN

l  Selection of expert FR engines: taking advantage of social context and social network context

o    Contact list of the photographer

o    Manually labeled photos

Fig.6. Construction of a social graph model using a contact list and personal photo collections. The thickness of the lines in Fig. 6(b) represents the strength of the social relationship between the current user and a contact. The larger the weight, the thicker the edge, and the closer the current user and his/her contact

l  Merging FR results

o    Multiple FR results from a set of heterogeneous FR engines: map incomparable scores onto a common representation photographer

o    Consider relevance score of associated FR engines

Fig.7. Method of merging FR results

3. Experiment

l  Retrieval of 547,991 personal photos from four volunteers and their contacts on Cyworld, a Korean online social network

Fig.8. Example images in Cyworld DB

Table1. Information of four volunteers in OSNs. (a) age, gender, contacts, years active for each volunteer. (b) the number of photos, tagging, and detected faces for each volunteer.



l  Observations

o    Large portion of face images belong to the photographer: numbers range from 13.4% for Volunteer 2 to 24.7% for Volunteer 1

o    Most of the face images belong to contacts of the photographer

  Numbers range from 73% for Volunteer 3 to 93% for Volunteer 4

  The identity of query face images not belonging to individuals enrolled in the contact list of a volunteer needs to be asked to the volunteer

o    Most of the face images only belong to a small number of contacts of the photographer: 91% of the query face images of Volunteer 1 belong to 28 contacts

Fig.9. Effectiveness of selecting FR engines

Fig.10. Effectiveness of annotating faces

l  Non-collaborative FR

o    Accuracy is measured by averaging the face annotation accuracy of all FR engines used to perform collaborative FR

l  Collaborative FR

o    Fusion of the multiple FR scores is either done using a Bayesian decision or majority voting

o    Take into account the relevance of a FR engine

o    Collaborative FR is much more effective than non-collaborative FR by virtue of a complementary effect caused by fusion of multiple FR scores


4. Conclusion

l  The collaborative use of multiple FR engines allows for improving accuracy of face annotation for personal photo collections shared on OSNs

o    A FR engine is highly specialized for the task of recognizing a small group of individuals

o    Query face images have a higher chance of belonging to people closely related to the photographer

l  Our collaborative FR framework is well-suited for use in promising decentralized OSNs

o    Each user in a decentralized OSN will have a personalized FR engine

o    These personalized FR engines will be shared by members of an OSN

l  Our collaborative FR framework is expected to enrich personal photo-based applications deployed in smart TV or smart phone systems connected to OSNs


* Contact Person: Prof. Yong Man Ro (ymro@ee.kaist.ac.kr)

: : : http://ivylab.kaist.ac.kr/image/demo/project_publication.gif

1.     Jae Young Choi, Wesley De Neve, Konstantinos N. Plataniotis, and Yong Man Ro, Collaborative Face Recognition for Improved Face Annotation in Personal Photo Collections Shared on Online Social Networks, IEEE Transactions on Multimedia, vol. 13, no. 1, pp. 14-28, 2011.

2.     Jae Young Choi, Wesley De Neve, Yong Man Ro, and Konstantinos N. Plataniotis, Automatic Face Annotation in Personal Photo Collections Using Context-based Unsupervised Clustering and Face Information Fusion, IEEE Transactions on Circuits and Systems for Video Technology, vol. 20, no. 10, pp. 1292-1309, 2010.