Scenario Test of Facial Recognition for Access Control Abstract William P. Carney Analytic Services Inc. 2900 S. Quincy St. Suite 800 Arlington, VA 22206 Bill.Carney@anser.org This paper presents research conducted using a scenario test of a facial recognition authentication system. The goal of the research was to test the hypothesis that a facial recognition authentication system could be integrated into an existing Closed Circuit Television (CCTV) system in an office, and that the system could perform an access control function at the levels of false acceptance and false rejection designated by the facial recognition vendor. Different lighting conditions and enrollment image qualities were tested, and ultimately the hypothesis was disproved by the resulting data. Included are descriptions of the experiments, results, and conclusions. 1 Introduction Biometric technologies provide a means for software systems to identify or authenticate individuals. They use identifiers, such as an individual s face, fingerprint, or iris scan, that are more difficult to counterfeit than conventional access cards and pin numbers. The uniqueness of biometric data, as well as its availability, makes biometric technologies suitable for access control systems. The uniqueness of some biometric identifiers has not yet been fully established, but that topic is out of the scope of this paper. Traditional access control systems combine two elements to control access to selected entryways: something you know - a PIN or password - and something you have - a badge or card. Biometrics can be used in access control systems to add a third element of control: something you are your personal biometric information. With each added element, the access control system becomes harder for an impostor to circumvent. Facial Recognition is an emerging technology used for both identification and authentication of individuals. Authentication, the comparison of an individual against a claimed identity, rather than identification, the comparison of an individual against an unknown identity, is the focus of this paper. There are two main steps in using a facial recognition access control system, enrollment and authentication. In enrollment, a series of facial images are taken of an individual and then processed by the system algorithms. A digital template is created for the individual based on the information gleaned from the facial images and stored in a database for later
comparison. During the authentication process, an individual presents a token, or key, claiming a valid identity. This token is used to index the database and retrieve the associated digital template. New images are taken of the individual, and a new temporary template is created which is compared to the previously retrieved template for the individual. If the confidence value returned by the system is equal or greater than a pre-determined level, or threshold, the individual is authenticated and granted access. 2 Purpose The purpose of the research conducted at ANSER s headquarters was to test one hypothesis: that a facial recognition authentication system could be integrated into an existing Closed Circuit Television (CCTV) system in an office, and that the system could perform an access control function at the levels of false acceptance and false rejection designated by the facial recognition vendor. Two assumptions about the technology were made. The first was that lighting differences during the enrollment and verification stages of the test would have an effect on the results; and the second was that the quality of enrollment images would also affect the results. The various stages of testing were built around the hypothesis and the two assumptions in order to attempt to gauge the performance of the system in terms of False Acceptance Rate (FAR) and False Rejection Rate (FRR). 3 System Concept This test represents a scenario test of a biometric access control system, i.e. the conditions were real world, but the system was not put in place in a true operational environment where it would control access to an entryway. Scenario testing allows the viability of the system to be tested without the possibility of jeopardizing security in the test facility. The system tested was built upon a software development kit (SDK) that benchmarked both an FAR and FRR of less than 1% at a threshold value of.62. These error and threshold values became the basis for comparison for the performance of the test system. Fifty ANSER employees volunteered to participate in this test. 4 Test Setup There were three stages to the test procedure: enrollment, true authentication tests, and false acceptance authentication tests. 4.1 Enrollment There were four sets of enrollment images captured for each subject, each comprising of five images. The first set of images was captured with a video camera (camera 1), and the following three sets were captured with a digital camera at different resolutions for each set. The images captured using the video camera were the lowest quality. The four
sets of images were then enrolled into the facial recognition system, and a digital template was created for each set and stored in the database. 4.2 True Authentication Each subject was authenticated by comparing images taken in three camera and lighting configurations against all four digital templates associated with that subject. The three configurations were: camera 1 with ambient lighting (normal corridor lighting), camera 1 with enhanced lighting (ambient lighting with incandescent lamp), and a second video camera (camera 2) with ambient lighting. For each template and configuration combination, a subject was authenticated in five trials. During each trial, a proximity card (the token) was scanned, the corresponding digital template was loaded by the system, the system captured one frame of video, attempted to find a face, and then match that face, if found, to the corresponding template in the database. If a match was found, the trial stopped. If a face was not found or a match did not occur, the system would retry with the next frame of video, up to a maximum of five frames per trial. 4.3 False Acceptance Authentication False Acceptance was tested using three digital templates for each subject and two camera configurations: camera 1 with ambient lighting and camera 1 with enhanced lighting. For each set of tests, each subject was authenticated a total of N 1 times. The process for authentication was the same as in the true authentication stage, with the exception that the proximity cards scanned during this stage did not correspond to the subject s template in the database. 5 Results True Authentication Test ENROLLMENT Video Camera Digital Camera Low Resolution Medium Resolution High Resolution Camera1 ambient lighting Percent Faces Found 75.6 81.2 94.1 81.2 Percent of Found Faces Verified 93.1 40.6 63.8 58 System Authentication 70.4 32.9 60 47.1 Avg Confidence of Verified Faces 0.79 0.684 0.673 0.657 Avg Confidence of Unverified Faces 0.519 0.576 0.578 0.581 False Rejection Rate (FRR) 29.6 67.1 40 52.9 Camera1 enhanced lighting Percent Faces Found 92.6 81.2 76.5 82.5 Percent of Found Faces Verified 92 71 92.3 93.9 System Authentication 85.3 57.6 70.6 77.5 Avg Confidence of Verified Faces 0.698 0.675 0.692 0.687 Avg Confidence of Unverified Faces 0.596 0.595 0.583 0.587 False Rejection Rate - FRR 14.7 42.4 29.4 22.5 Camera2 ambient lighting Percent Faces Found 70.1 90.6 95.3 92.9 Percent of Found Faces Verified 43.6 97.4 96.3 97.5 System Authentication 30.6 88.2 91.2 90.6 Avg Confidence of Verified Faces 0.677 0.675 0.674 0.671 Avg Confidence of Unverified Faces 0.579 0.594 0.588 0.604 False Rejection Rate - FRR 69.4 11.8 8.8 9.4 Table 1: True Authentication
False Acceptance Authentication Test ENROLLMENT Video Camera Digital Camera Low Resolution High Resolution Camera1 ambient lighting Percent Faces Found 86.8 91.7 93 Percent of Found Faces Verified 24.9 0.007 0.004 System Authentication (FAR) 21.6 0.006 0.003 Avg Confidence of Verified Faces 0.645 0.633 0.636 Avg Confidence of Unverified Faces 0.563 0.536 0.537 Camera1 enhanced lighting Percent Faces Found 89 84.6 87.4 Percent of Found Faces Verified 12.1 1.48 2.81 System Authentication (FAR) 10.8 1.25 2.5 Avg Confidence of Verified Faces 0.635 0.631 0.625 Avg Confidence of Unverified Faces 0.559 0.544 0.546 6 Analysis Table 2: False Acceptance Authentication The first set of data to observe is the FAR (Table 2) and FRR (Table 1) for authentication trials of video enrollment at camera 1. These were the control set of trials where the system was tested without changes to lighting and using basic enrollment images. This set of trials resulted in an FRR of 29.6% and an FAR of 21.6%, both extremely high, and unacceptable for access control systems. With the addition of the enhanced lighting for authentication, the numbers decreased dramatically to an FRR of 14.7% and FAR of 10.8%. However, while these numbers are markedly improved, they are still much higher than the goal of <1% for the system. In fact, none of the sets of trials resulted in an FRR close to the aforementioned goal. On the other hand, authentication using digital camera images at camera 1 with ambient light produced FAR well under 1%. Unfortunately, the FRR for these trials were exceedingly high as shown by authentication of subjects using high resolution digital camera enrollment images under ambient light which resulted in an FAR of.003% and an FRR of 52.9%. 7 Conclusions The results of the test serve to disprove the hypothesis rather soundly. In certain trials, the system was able to produce low error rates for false acceptance, but it never was able to produce low rates for both false acceptance and false rejection for the same enrollment sets. The results also serve to prove the assumptions correct that were made prior to the test concerning lighting and enrollment image quality, although the effects of these variables were inconsistent. Enhanced lighting improved false rejection rates for all sets and improved false acceptance rates for video enrollment, but increase false acceptance rates slightly for digital camera enrollments. Higher quality enrollment images resulted in the lowest FAR, and also the lowest FRR under camera 2, but resulted in very high FRR under camera 1. While the test was centered on gauging the system performance based on FAR and FRR, there were other interesting lessons learned as a result of performing this test. One
problem that occurred during the test was difficulty in enrollment for some subjects. Some subjects were able to enroll in only one or two attempts, while others needed several attempts to be enrolled. Many factors can affect the ability to enroll, most notably distance from the camera, head position, and eye position. In a true operational environment, dealing with users that have difficulty enrolling or who cannot enroll at all would be a major hurdle in implementing a biometric access control system. A second point to be made is that due to the configuration of the existing CCTV system, subjects had to be height screened before testing could occur. Subjects exceedingly tall or short were unable to be seen in the viewing range of the system and could not participate. For this test, subjects needed to be between 5 3 and 6 3 in order to participate. Again, an operational biometric access control system would need to address height issues for all users in an operational setting. A third point of note is that although this test was concerned with the performance of the facial recognition system over all subjects, certain subjects stood out from the rest. In particular, while the FRR for authentication at camera 1 with ambient lighting against video enrollment was 29.6%, there were several individuals who were correctly authenticated every time. Under the same authentication conditions, the FAR was 21.6%, but there were several users whom the system never falsely accepted, as well as one user who was falsely accepted as over 50% of the identities in the system. These points of data further emphasize the need of an operational biometric access control to perform at acceptable levels for all users, and not just for the average user. A final point to make is that the size of the subject set serves to further emphasize the need for a biometric access control system to perform properly for all users. Had the data shown that the system worked well for all subjects involved in the test, it would be unwise to assume that the system could work for all subjects in a much larger sample set. However, the fact that the system performed poorly on a small sample set suggests that the problems that occurred during the scenario test would likely exist on a much larger sample set. The test performed at ANSER served to show that a facial recognition access control system cannot be implemented into an environment and be guaranteed to perform at acceptable levels. Environmental issues such as lighting, enrollment image quality, and quality of hardware are essential factors that need to be addressed in an operational facial recognition system. Furthermore, this test has proven the value of scenario testing being conducted on biometric systems before they are integrated into operational systems. In environments where maintaining security is a paramount concern, it is inadvisable to install a system that has not been tested under similar conditions as the operational environment. 8 Acknowledgements This research has been partially supported by the National Institute of Justice; grant numbers 97-LB-VX-K025 and 98-LB-VX-K021, as part of the work on advanced facial
recognition and intelligent software agents. The opinions stated herein are solely those of the author, and not necessarily those of the National Institute of Justice. Thanks to Corey Wineman and Sanjeev Kapoor for work on the test system. Thanks also to Ron Turner, PhD, Jeffrey Goldberg, PhD, Mark Spooner, and Bernard Kellett for editing assistance. 9 References [1] William P. Carney, Jeffrey L. Goldberg, Mark A. Spooner, Anatoli M. Peredera, Sanjeev K. Kapoor, Facial Recognition for Access Control, CISST 03, June 2003. 10 Appendix System Components: PC Components Cameras and Video Equipment Door Controller Components Windows 2000 PC with 500 MHz Pentium III processor and 384MB RAM FlashBus MV Pro video capture card Access Control Software Pelco camera model DF5HD, wall mounted (camera 1) Pelco pan tilt zoom camera model Spectra II,DD5BC (camera 2) Kodak DC260 digital camera at 768X512 (low), 1152X768 (medium), and 1536X1024 (high) resolutions 21 Pelco Color Monitor Model PMC21A Pelco Genex series Multiplexer model MX4016CD Pelco Multiplexer Keyboard Model KBD4000 600 Watt quartz standing incandescent lamp with umbrella HID Proximity cards, Proxcard II, 26 bit Weigand Mullion mount proximity reader 26 bit Single reader interface board Enclosure with power supply for Lenel controllers 16 door controller unit Securitron 600 lb. Magnetic door lock Model M32-12