Yitu Technology won first place in prestigious global facial recognition benchmark test
SILICON VALLEY, Calif., Sept. 5, 2017 /PRNewswire/ -- In the recently released results of Face Recognition Vendor Test, a prestigious international benchmark, Yitu Technology, a company from China, won first place in four different categories, overtaking the Russia company Vocord, a long-established international security brand that beat Google in various facial recognition benchmarks for years, and Morpho, a well-known French biometrics company.
Face Recognition Vendor Test (FRVT), is a benchmark organized by National Institute of Standards and Technology (NIST), which serves as the official guideline for U.S. government purchases. The test results are the golden standards of global security industry in practice.
The test datasets of FRVT are all of real-world scenarios provided by the U.S. Department of Homeland Security, e.g. massive number of photographs taken by border control agencies and in criminal investigations. Compared to other popular tests setup by academia, such as LFW and MegaFace, FRVT is more practical and at a larger scale. Not only is the test dataset scale in the order of millions, but also it is not available to the public, thus, results in effective prevention against algorithmic over-fitting or any manner of unfair practices.
Table 1: Comparison on the test scenarios, datasets and key factors of facial recognition.
Administered By Data sources Dataset size Testing data Performance metrics
--------------- ------------ ------------ ------------ -------------------
LFW University of Massachusetts Internet Celebrities 10,000 (easy to over-fit Testing data is public; difficult Recognition performance results
data) to ensure training and test data in 1/1000 false match rate
training independence making it difficult to determine
difference between algorithms
--- --------------------------- -------------------- ------------------------- ---------------------------------- --------------------------------
MegaFace University of Washington Internet Celebrities 1,000,000 Testing data is public; difficult First hit rate from over a
to ensure training and test data million data points allows for
independence comparing differences among
algorithms
-------- ------------------------ -------------------- --------- ---------------------------------- -------------------------------
National Institute of Standards and Official data from US immigration 1,000,000 Testing data is not public (blind Largest scale for testing,
Technology and criminal investigations test) 1/10,000,000 false match rate
(sample taken from pool of over
10 billion sample pairs)
FRVT
----
Table 2: Comparison on the results of renowned brands on three test datasets (Hikvision did not provide data)
Yitu Technology Vocord Tencent Youtu Dahua Baidu Megvii Sensetime
--------------- ------ ------------- ----- ----- ------ ---------
FRVT 95%, 1st place 93.5% 2nd place
---- -------------- ---------------
MegaFace 91.7% 1st place 83.3% 2nd place
-------- --------------- ---------------
LFW 99.8% 2nd place 99.78% 4th place 99.77% 7th place 99.5% 99.53%
--- --------------- ---------------- ---------------- ---- -----
Labeled Faces in the Wild (LFW), is collected and maintained by the University of Massachusetts. he test dataset scale is in the order of ten thousand. A number of companies from China, including Baidu, Tencent, Dahua, Megvii, and SenseTime, achieved results of over 99.5% accuracy.
MegaFace is a public facial dataset collected and maintained by the University of Washington, which is 100 times larger than LFW and more capable of distinguishing performance differences of algorithms developed by participants.
The long-established Russia company Vocord won first place, with 91.7% accuracy, which is 8% ahead of Tencent, which ranked second.
In the highest class of competition, the advantages in training dataset and computations do not guarantee the best algorithmic performance.
FRVT is organized by NIST
The test results have the highest credibility, which originate from the rigorous and comprehensive.
1. Test datasets are from real-world scenarios, meaning good results imply
immediate straight deployment of the technology.
2. The dataset is in the order of 1 million out of 10 billion possible
samples.the evaluation processed with such a large-scale test dataset
gives consistent results on algorithm performance, thus distinguishes the
leading companies in the field.
3. The test dataset is not available to the public (blind test), thus
fairness is assured. Due to the unavailability of the test dataset, the
participants are not able to use them as a training set, thus
over-fitting, or any other unfair advantages were effectively avoided.
Therefore, FRVT reflected fair results over LFW and MegaFace.
4. FRVT adapts professional and objective approaches. FRVT provides detailed
evaluations of recognition performance, such as, the speed of the models,
the size of characteristics databases, comparative thresholds, etc. on
different areas like background, ethnicity, age and gender.
According to the latest official report, Yitu ranked No.1
The result is 95.5% accurate, subjected to the false match rate of ten-millionth (based on 10 billion samples). This is 2% higher than the Russia company, Vocord, whose accuracy is only 93.5%.
This is the ever-best result from the most prestigious public evaluation based on the large scale dataset, which shows a significant increase from 2014 when NEC, a Japanese company, ranked first.
FRVT evaluates recognition performance in details concerning different scenarios, ethnicities, genders and ages. As described in the official FRVT report, its test dataset covers real-world scenarios such as border control, ID passes and city-level security.
Yitu ranked No.1 in four major scenarios.
The algorithm developed by Yitu was consistent in its performance in detailed ethnicities and gender tests. However, as the difficulty increases, the algorithms submitted by its rivals showed a drop in recognition accuracy. The results justify the adaptability of Yitu's algorithm on different datasets of various real-world scenarios. For example, Yitu's algorithm performed well despite current issues with darker skin tones.
The report as of June 19, 2017:
https://www.nist.gov/sites/default/files/documents/2017/06/20/frvt_report_2017_06_19.pdf
The unbelievable increase of Artificial Intelligence
Over the last for years the key testing factors and false match rates have dropped from one-thousandth to ten-millionth. The accuracy has increased ten thousand times. It could be foreseen that recognition performance comparison subject to the key test factor of false match rates of thousand-millionth, one-billionth, or even ten-billionth will emerge very soon.
This would be another AI recognition power growth of thousand times.
We can imagine that high performance facial recognition based on super-large-scaled comparison, as large as one in one-billion, with super-low false match rate of one-billionth, will unleash this technology to wider range of daily settings.
Facial recognition at all levels, from a single building, to a city, or metropolitan areas, to even a whole nations, is becoming possible.
For more information contact:
Li Jingyi
+86-139-1082-5111
jingyi.li@yitu-inc.com
Han Yang
+86-137-7109-2666
yang.han@yitu-inc.com
View original content:http://www.prnewswire.com/news-releases/yitu-technology-won-first-place-in-prestigious-global-facial-recognition-benchmark-test-300513646.html
SOURCE YITU TECH
