|
|
Collaborative Face Recognition for Face Annotation Applications in
Personal Photos Shared on Online Social Networks |
|
|
|
1. Introduction
l Online Social
Networks (OSNs)
o The number of
personal photos in OSNs keeps significantly growing due to
¡× Easy-to-use multimedia
devices and online services
¡× Cheap storage
and bandwidth
¡× An increasing
number of people going online
o Sharing of
personal photos in OSNs
¡× Associated with
their photographer
¡× Being
broadcasted out to the online contacts of the photographer
¡× Joining a
collection of billions of other photos
o Problem: Digital
information overload (requirement to automatically organize photos in OSNs
Fig.1. Image
Sharing in OSNs
l Face annotation
or name tagging
o Allows
person-based organization and retrieval of photos
o Considerable
practical interest due to high commercial potential
o Automatic face
detection and face recognition (FR)
Fig.2. Face
Detection from Personal Photos and Process of Face annotation and Retrieval
l Architecture of
OSNs
o Centralized
¡× Centralized
server with all user data (e.g., contact list, photos)
¡× Personal several
problems (information silos, ownership of data, and privacy issues)
o Decentralized
¡× Attracting more
and more interest
¡× Present several
advantages
¡¤
Each user has a personal server containing his/her user
data
¡¤
Giving users more control over data ownership and privacy
Fig.3.
Architecture of OSNs
l Research
Motivation:
o Limitation of
traditional FR:
¡× Appearance-based
FR solutions still come with a low accuracy
¡× Restricted to be
developed using centralized FR framework relying on a single FR engine
o Decentralized
OSNs
¡× Each user will
have a personalized FR engines, optimized for recognizing a small set of
individuals in question
¡× Sharing FR
engines between a user of the OSN and his/her contacts
o Development of a
novel collaborative FR framework
¡× Social context
in personal photos
¡× Social network
context in OSNs
¡× Use of multiple
FR engines
l Research Goal:
Given a photographer, the goal is to use FR engines of other users for face
annotation in personal photos of the given user
Fig.4. Proposed
framework of collaborative FR
2. Collaborative Face Recognition
l Research
challenges:
o How to select
expert FR engines?
o How to merge
multiple FR scores in a single decision?
o How to deal with
heterogeneous FR engines? (FR engines using different FR techniques)
Fig.5. Proposed
collaborative FR framework in an OSN
l Selection of
expert FR engines: taking advantage of social context and social network
context
o
Contact list of the photographer
o
Manually labeled photos
Fig.6.
Construction of a social graph model using a contact list and personal photo
collections. The thickness of the lines in Fig. 6(b) represents the strength of
the social relationship between the current user and a contact. The larger the
weight, the thicker the edge, and the closer the current user and his/her
contact
l Merging FR
results
o
Multiple FR results from a set of heterogeneous FR
engines: map incomparable scores onto a common representation photographer
o
Consider relevance score of associated FR engines
Fig.7. Method of
merging FR results
3. Experiment
l Retrieval of
547,991 personal photos from four volunteers and their contacts on Cyworld, a Korean online social network
Fig.8. Example
images in Cyworld DB
Table1. Information of four volunteers in OSNs. (a) age, gender, contacts, years active for each volunteer. (b) the number of photos, tagging, and detected faces for each
volunteer.
(a)
(b)
l Observations
o Large portion of
face images belong to the photographer: numbers range from 13.4% for ¡®Volunteer
2¡¯ to 24.7% for ¡®Volunteer 1¡¯
o Most of the face
images belong to contacts of the photographer
¡× Numbers range
from 73% for ¡®Volunteer 3¡¯ to 93% for ¡®Volunteer 4¡¯
¡× The identity of
query face images not belonging to individuals enrolled in the contact list of
a volunteer needs to be asked to the volunteer
o Most of the face
images only belong to a small number of contacts of the photographer: 91% of
the query face images of ¡®Volunteer 1¡¯ belong to 28 contacts
Fig.9.
Effectiveness of selecting FR engines
Fig.10.
Effectiveness of annotating faces
l Non-collaborative
FR
o Accuracy is
measured by averaging the face annotation accuracy of all FR engines used to
perform collaborative FR
l Collaborative FR
o Fusion of the
multiple FR scores is either done using a Bayesian decision or majority voting
o Take into
account the relevance of a FR engine
o Collaborative FR
is much more effective than non-collaborative FR by virtue of a complementary
effect caused by fusion of multiple FR scores
4. Conclusion
l The
collaborative use of multiple FR engines allows for improving accuracy of face
annotation for personal photo collections shared on OSNs
o A FR engine is
highly specialized for the task of recognizing a small group of individuals
o Query face
images have a higher chance of belonging to people closely related to the
photographer
l Our
collaborative FR framework is well-suited for use in promising decentralized
OSNs
o Each user in a
decentralized OSN will have a personalized FR engine
o These
personalized FR engines will be shared by members of an OSN
l Our
collaborative FR framework is expected to enrich personal photo-based
applications deployed in smart TV or smart phone systems connected to OSNs
|
* Contact Person: Prof. Yong Man Ro (ymro@ee.kaist.ac.kr) |
1. Jae Young Choi, Wesley De Neve, Konstantinos N. Plataniotis, and Yong Man Ro, ¡°Collaborative Face Recognition for Improved Face Annotation in Personal Photo Collections Shared on Online Social Networks,¡± IEEE Transactions on Multimedia, vol. 13, no. 1, pp. 14-28, 2011.
2. Jae Young Choi, Wesley De Neve, Yong Man Ro, and Konstantinos N. Plataniotis, ¡°Automatic Face Annotation in Personal Photo Collections Using Context-based Unsupervised Clustering and Face Information Fusion,¡± IEEE Transactions on Circuits and Systems for Video Technology, vol. 20, no. 10, pp. 1292-1309, 2010.