The recent sharp increase in terrorist attacks has made the security of public facilities such as airports, restaurants and hotel lobbies a global issue. The face image in a passport is an only data to be used for personal authentication worldwide, regardless of nationality. In addition, the face image recognition offers a unique advantage compared to other personal authentication technologies because it enables to authenticate people from a distant location such as personal identification through surveillance cameras. NEC joined the Face Recognition Vendor Tests of the U.S. National Institute of Standards and Technology (NIST) in 2009 and gained the top ranking three times consecutively. This paper introduces the NEC video face recognition technology for use in video surveillance.

1.Introduction

A sharp increase in crime on a worldwide scale has increased the opportunities for using the face recognition technology. As shown in Fig. 1, NEC started R&D into face recognition technology in 1989 and commercialized the face recognition engine “NeoFace” in 2002. Since then, we joined the Face Recognition Vendor Tests of the U.S. National Institute of Standards and Technology (NIST) held in 20091), 20102) and 20133) and gained the world-top ranking in three consecutive test years. Our products have already been adopted in more than 100 systems by forty countries worldwide. These include those of the world’s major organizations such as New York JFK Airport, Australian governmental institutions and major Brazilian airports4). Featuring both high convenience and high authentication accuracy, our face recognition technology is actively expanding the situations of use, e.g., it was used at 50 music concerts and sports events in Japan in 2015 for audiences5) (Fig. 2), numbering more than 300,000.

For face recognition surveillance purposes, we released a product named “NeoFace Watch” in 20126). Fig. 3 shows typical scenarios of face recognition systems for surveillance purposes. This technology makes it possible to find each person registered in the system in video images shot in various situations including in a passageway, elevator and staircase. The wide variety of usages for which it can be applied include scenarios such as crime investigation, the straying of children and dementia patients.

Below in section 2 we describe the video face recognition technology proposed by NEC. In section 3 we go on to report on the results of recognition performance evaluation using low-resolution images that were conducted at the NIST. We then introduce some actual examples of applications in section 4, before presenting a short conclusion to the paper.

2.Introducing the Video Face Recognition Technology

Fig. 4 shows the video face recognition processing flow. The system first detects a face part in a captured image, and then extracts feature points in the face, including the pupils, subnasal points and mouth corners. Finally, the system extracts the face feature quantity, matches it with the feature quantity data registered in a database and identifies a person as one of the registered persons with a degree of resemblance higher than a certain threshold level.

The following subsections discuss details of each technology (Fig. 5)7).

2.1 Face Detection Technology

For face detection, rectangular areas that match the face are extracted by sequentially searching face areas, starting from the edge of the image. The Generalized Learning Vector Quantization algorithm, which is based on the NEC-original pattern recognition technique called the Minimum Classification Error criterion, is used to recognize whether areas are of face areas or not. This procedure enables a fast and accurate face detection function.

2.2 Feature Point Extraction Technology

The feature point extraction technology finds the positions of face feature points such as a pupil, subnasal point and corner of the mouth. Brightness patterns around the feature are used to find the optimum position, while at the same time a facial shape model is used to constrain the alignment of features, thereby enabling precise estimation of their positions.

2.3 Face Matching Technology

In order to identify whether or not the person with the captured face image is a registered person, the face matching process normalizes the face position using the obtained feature points. The captured face image and the candidate image are then collated. After extracting face features, such as the shapes and tilts of eyes and nose, the optimum feature for identifying the person is selected using the multi-dimensional feature recognition method. This procedure enables a robust personal identification solution that is unaffected by changes due to aging or other factors.

3.Results of Performance Evaluation with Low-Resolution Face Images

What is important for the implementation of the face collation used for surveillance purposes is the possibility of collation using low-resolution images. When the person is in a position close to the surveillance camera as shown in Fig. 6, the collation is easy because a face image with high resolution can be captured. However, when the person is closer to the camera, collation becomes difficult because the elevation angle increases and the face is often hidden by a large area due to the hair coming over the face. In addition, from a practical viewpoint also, the area covered by each camera is narrowed so that this layout is unsuitable for surveillance purposes. On the other hand, when a person is at a certain distance from the surveillance camera, the elevation angle decreases so the face can be captured from a more or less frontal direction. As the area covered by each camera increases, this aspect is effective for surveillance. However, the fact that the captured face image has low resolution requires the face collation technology to adopt suitable countermeasures. In other words, when the low-resolution face collation technology can be implemented at a practical level the advance toward the realization of effective face recognition for surveillance is enabled.

Fig. 7 shows the relationship between the face resolution and collation error rates, which is quoted from the evaluation results by the NIST in 20118). The horizontal axis represents the inter-eye distance (number of pixels between the centers of the two eyes) and vertical axis the collation error rate. The engines of tested organizations other than NEC showed a disadvantage against low resolution, presenting gradual performance degradation when the inter-eye distance was less than 96 pixels. On the other hand, although NEC’s face recognition engine achieved high resolution and low collation error rates. It also presented little performance degradation even at as small an inter-eye distance as 24 pixels, thereby proving that the technology is very effective for low-resolution images.

As seen above, NEC’s face recognition engine is capable of highly accurate recognition even when surveillance cameras are in a relatively distant location and it is therefore suitable for surveillance purposes.

4.Applications of Video Face Recognition Systems

NEC’s face recognition systems are currently in use by more than 100 systems in 40 countries worldwide. The original purpose was the personal authentication using passports, but surveillance applications are also increasing gradually.

4.1 Face Recognition Systems at 14 Brazilian Airports

Persons entering and exiting Brazil have been increasing because of the world-scale sport events held in that country. However the subsequent growth of illegal behaviour at customs has necessitated the more efficient surveillance of passengers. In July 2015, the Department of Federal Revenue of Brazil introduced face recognition systems in the customs areas of fourteen major international airports in Brazil (Fig. 8)9). These systems apply face recognition to those passengers passing through customs and identify any suspicious persons based on a list of those who have previously been charged with illegalities. They have thereby improved the efficiency of customs operations at the airports, also contributing to the security and improved customer relations at public facilities by flexibly meeting the diversified needs both of the government and of enterprises.

4.2 Biometric Identification System for an Australian Governmental Agency

CrimTrac, an Australian federal governmental agency has introduced a biometric identification system based on the face recognition and fingerprint identification technologies10). This system allows the databases with 12 million facial images currently possessed by the police, to be shared by the relevant Australian authorities for use in crime investigations and border security. A typical assumed application is to capture the facial image of a suspicious person with a security camera and identify such a person by collating the image with the CrimTrac database.

4.3 Video Face Recognition Solution for Indian Hotel Group Lemon Tree Hotels

The Indian hotel group Lemon Tree Hotels introduced a face recognition system in 201411). This solution is used for customer security and the improvement of services at the Lemon Tree Premier Hotel, Delhi Airport. This is a hotel located near the Indira Ghandi International Airport in New Delhi Aerocity. The system captures the face of each customer and collates images in real time with the criminal database provided by the police. In addition, the images are also collated with the VIP guest list of the hotel.

5.Conclusion

In the above, we discussed the video face recognition surveillance procedure from the aspect both of technology and applications. Now that concerns about security are increasing on a worldwide scale the face recognition technology is becoming an essential security infrastructure that is considered to be indispensable for the secure running of a country.

Based on the world’s top level image recognition technology that was cultivated via our experience in developing the face recognition technology, we will continue the R&D so that face recognition can contribute to the creation of a safe, secure world. This will be achieved by targeting a leap toward technology that can handle large-scale, mission-critical situations effectively and can offer high level applications to suit various scenarios.

Reference

1) P. Jonathon Phillips: Still Face Challenge Problem Multiple Biometric Grand Challenge Preliminary Results of Version 2, MBGC 3rd Workshop, 2009

2) NIST Interagency Report 7709: Report on the Evaluation of 2D Still-Image Face Recognition Algorithms

3) NIST Interagency Report 8009: One-to-Many Face Recognition Report

7) IMAOKA, “The World’s Best Face Recognition System to Achieve Safety and Security in Our Societ”, NEC Technical Journal, Vol. 9, No.1, pp.48-51, January 2015

8) NIST Interagency Report 7830: Performance of Face Recognition Algorithms on Compressed Images

* Results shown from the MBGC2009, MBE2010 and FRVT2013 (Face Recognition Vendor Test 2013) do not constitute endorsement of any particular product by the U.S. Government.

Authors’ Profiles

IMAOKA Hitoshi
Research Fellow
Data Science Research Laboratories