March 28, 2019

High-speed-camera object recognition technology combines high-speed cameras with image recognition technology to perform real-time recognition without having to stop the objects, even when moving at high speed. In this interview, two developers discuss the details of this innovative technology.

Enabling the recognition of moving objects in the flow of the production line without setting up special processes

Achieving authentication with speed and high accuracy

― What sort of technology is used with the high-speed cameras?

Yokoyama: In image recognition of moving objects, the three major types of processing performed are imaging, tracking, and recognition. With respect to imaging and tracking, high-speed cameras play a major role. For example, if 1000 images per second are captured, the camera can reliably capture a moment that is suitable for the recognition process, even when the object is moving. In addition, the motion is reduced to an extremely slow speed, so that the target can be tracked using only lightweight and high-speed processing. The Ishikawa Senoo Laboratory at Tokyo University, which we collaborated with to jointly develop our new technology, is pursuing cutting-edge research in vision technology that utilizes the features of these types of high-speed cameras.

Namiki: In addition, NEC possesses high technical capabilities in recognition. In particular, we have come to lead the world in the field of image recognition technology and we have, for example, long achieved results that demonstrate the world's best accuracy and speed in face recognition.* As a result, there was great potential in terms of integrating this image recognition technology with sensing that uses high-speed cameras.
However, high-speed cameras that operate at 1000 frames per second produce a large volume of images, which inevitably requires a vast amount of processing and takes a long time. To resolve this issue, we developed technology for selecting images that are suitable for recognition, which is at the core of our high-speed-camera object recognition technology, along with image recognition technology that achieves significant weight reduction while maintaining accuracy.

Yokoyama: The technology for selecting suitable images for recognition instantly judges and selects images that are appropriate for image recognition, from the large volume of images captured by the high-speed camera. Using the information that is obtained when the images are captured, such as the amount of movement of the object, images are selected according to criteria such as the degree of focus and blurring, and the presence of clearly defined edges. Images are narrowed down to about one out of several dozen. Unlike when images are extracted randomly, this makes it possible to extract multiple images that all are appropriate for recognition, so that the accuracy of image recognition is improved. On the other hand, if blurred images are selected, noise occurs and the accuracy is reduced.
One way to reduce the amount of data is to lower the image resolution, but that sacrifices the detailed information that is the basis of high-accuracy judgment. Selecting the appropriate images contributes greatly to improving the accuracy.

Namiki: In the recognition process, recognition is performed several times using multiple extracted images. Even in a single recognition operation, methods such as deep learning can be used to enable high-accuracy judgment. However, this increases the computation volume, which inevitably leads to time limitations. In addition, if one selected image is an extremely confusing image that is difficult to judge, it can lead to misrecognition. If, on the other hand, the processing is performed multiple times and judgment is based on the majority results, a higher accuracy can be achieved than with a single judgment, even when the recognition only has a fair degree of accuracy. In the recognition technology that we developed, this method is used to achieve speedy recognition. By the way, although I mentioned a "fair degree of accuracy", our technology does not perform rule-based recognition that simply finds a proper threshold for color or shape recognition. As a result, when judging whether products are acceptable or not, for example, there is a lower probability of over-functioning that leads to separating out good-quality products that are falsely identified as defective. Our technology uses machine learning to achieve recognition with high accuracy and flexibility, while also considering the speed.

* Press release - March 16, 2017

"NEC’s Video Face Recognition Technology Ranks First in NIST Testing"

Applications in factory in-line inspection and product traceability

― What types of applications are being considered?

Yokoyama: We are now having meetings with customers in a variety of industries and businesses to discuss verification testing.
At present, we are considering applications for use in manufacturing plants. In recent years, the global trend is shifting away from conventional small-variety large-quantity production toward large-variety small-quantity production (multi-product variable production), which flexibly responds to customer needs. With the diversification of manufacturing processes, the inspections between processes are becoming increasingly important. However, the inspection patterns are becoming more complex, leading to major issues such as how to secure the human resources for inspection, or how to streamline the process of changing the setups. With this in mind, we believe that our high-speed-camera object recognition technology can be practically applied in systems that recognize and automatically sort products based on their external appearance, even in mixed lines with multiple products. This achieves in-line inspection without having to set up separate inspection processes for each product. This is one of the goals that we are considering now.
When I first entered NEC, I was involved in factory production innovation and supply chain management. I was transferred to the laboratories at the same time that this project started, which was about one and a half years ago. I think that my experience gives me a good understanding of the issues facing manufacturing sites, and I believe that our technology can be used as a high-speed and high-accuracy automation tool that works effectively to resolve these issues and improve throughput.

Namiki: Speaking of the manufacturing industry, I believe the technology can also be successfully applied in the pill manufacturing process. Characters are printed on pills, but this printing can cause scratches or chips. Our technology can effectively pick out these types of detailed defects. In addition, I think that in the future, the technology can be applied in the recovery of rare metals from devices such as smartphones. This is generally called "urban mining." At present, to recover rare metals, people must disassemble each product one at a time by following the assembly process in reverse. If the devices are ground down into small pieces, the desired parts cannot be identified by their shape. However, if high-speed-camera object recognition technology is combined with other technology, I think that the targeted rare metals can be extracted even when devices are ground down into small pieces.

Yokoyama: An essential aspect of our technology is its ability to eliminate the active operations and processes that are required in image recognition, so that it becomes invisible and enables passive authentication. The technology has great potential in a variety of areas, such as in biometric authentication, so I hope that it will be actively deployed in a wide range industries and businesses.

  Furthermore, part of our technology is based on the results of “Realtime IoT System Using High-speed Vision Sensor Network and Development of Its Application Technology” in the “Crosscutting Technology Development Project to Promote IoT” at the New Energy and Industrial Technology Development Organization (NEDO).

Go back to Featured Technologies