Skip to content. | Skip to navigation

Personal tools

Navigation

You are here: Home / Testing / Competition on face recognition in mobile environment using the MOBIO database

Competition on face recognition in mobile environment using the MOBIO database

In the context of BEAT project, the Biometric group at the Idiap Research Institute is organizing the second competition on face recognition for the 2013 International Conference on Biometrics (ICB-2013) to be held in Madrid, Spain on June 4-7, 2013. Researchers in biometrics are highly invited to participate in this competition. This will help them to evaluate the progress made in the last couple of years.

The competition will be carried out on the MOBIO database. MOBIO is a challenging bimodal (face/speaker) database recorded from 152 people. The database has a female-male ratio of nearly 1:2 (52 females and 100 males) and was collected from August 2008 until July 2010 in six different sites from five different countries. The images provided by the database were captured with cameras of mobile phones and a laptop computer.

More technical details about the MOBIO database can be found on its official webpage: https://www.idiap.ch/dataset/mobio


http://atvs.ii.uam.es/icb2013/images/icb2013.png

 

 

Particularity of MOBIO database

The MOBIO database is designed as to provide data that was recorded in a natural environment, i.e., using mobile devices. Hence, algorithms that perform well on this database are highly probable to be suitable for other real-world applications that do not require a predefined image recording setup.

In opposition to many of the well-known facial image databases, the MOBIO database provides unbiased face verification protocols, one for male and one for female clients. These protocols partition the clients of the database in three different groups:

  • a Training set: used to train the parameters of your algorithm, e.g., to create the PCA projection matrix for an eigenface-based algorithm
  • a Development set: used to evaluate hyper-parameters of your algorithm, e.g., to optimize the dimensions of the PCA matrix or the distance function
  • an Evaluation set: used to evaluate the generalization performance of your algorithm on previously unseen data

 

Development and Evaluation set are again split into images that are used to enroll client models, and probe images to be tested against all client models. In opposition to many other protocols, here several images per client are used to enroll a model.

The following table gives a detailed overview about the partition of the database between Training set, Development set, and Evaluation set, including the number of clients (identities) in the sets, the number of files used for enrollment and probe, and the number of scores that needs to be computed for the experiments:

Training setDevelopment setEvaluation set
Protocol Clients Files Clients Enrollment Probe Scores Clients Enrollment Probe Scores
MALE 37 7104 24 120 2520 60480 38 190 3990 151620
FEMALE 13 2496 18 90 1890 34020 20 100 2100 42000
(total) 50 9600 42 210 4410 94500 58 290 6090 193620

 

Particularity of the evaluation

The particularity of the evaluation is the restriction of using defined evaluation protocol for all participants. This gives a more appropriate and objective comparison between different systems.

Since the evaluation is done on data recorded on mobile devices, the processing time and the memory footprint of the algorithms are given special attention. For this reason, participants are invited to provide more details about these details at all different stages of their system (a form will be sent to participants at evaluation time).

    This competition is focused on evaluating face recognition systems rather than face detection systems. Hence, the database provides hand-labeled eye locations for all images of the database. If your algorithm does not make use of these positions, please add a note into your system description (see below).

     

    Downloading the data

    The data for the competition can be downloaded from the download section of the original MOBIO website. This will require an EULA to be signed. After getting access, you should download the file "ICB2013-train_dev_image.tar.gz", which contains both the image data and the file lists defining the evaluation protocol.

    For an example, how to use these file lists, we provide a baseline PCA+LDA system, which you can download here. The baseline script is written in python. To run the baseline will require to have installed the open source library Bob (only avaliable for Linux and MacOS based operating systems, we are working on a MS Windows version of Bob). When you put the "baseline.py" file to the same directory where you extracted the MOBIO data, the script should run out of the box, otherwise you might have to specify some command line options (see "baseline.py --help" for details).

    If you have trouble with either downloading the data or running the baseline script, please feel free to contact us.

     

    Important Dates

    At the beginning of the competition, participants will be provided with the training list, the list of the hand-labeled eye positions, and the enrollment and probe lists of the Development set. Additionally, the source code of a baseline face recognition system will be published to give an example on how to use these lists. At the evaluation time (March 1, 2013), the equivalent lists of the Evaluation set will be made available to the participants.

    At the end of the competition (March 15, 2013), the authors are invited to deliver the resulting score files of the Development and the Evaluation set, and give a short description of their system(s). These results will be published in a conference paper at the ICB-2013.

    Here is a summary of the important dates of the evaluation:

    Registration DueJanuary 14, 2013
    Availability of Training and Development sets
    January 14, 2013
    Availability of Evaluation set
    March 1, 2013
    Submission of the Results + System description
    March 15, 2013
    Publication of the Results at ICB-2013 April 8, 2013

     

    Contacts

    Manuel Günther Manuel.Guenther {at} idiap.ch
    Sébastien Marcel Sebastien.Marcel {at} idiap.ch
    This is themeComment