Recently in the manufacturing industry, there is an increasing trend to collect and analyse the big data of the history of manufacturing and using objects so as to connect those results to production innovations and maintenance & inspection services. Individual recognition is fundamental for the acquisition of data of the individual objects. Nevertheless, the traditional methods of identification by attaching barcodes or RFID tags to objects are not capable of recognizing the individual electronic components that are reducing in size, or the precision machines or materials that do not accept surface processing. This paper introduces applications of the Fingerprint of Things in the individual recognition and IoT. These enable the identification of individual parts of even uniformly manufactured industrial products by image recognition of the finely depicted surface patterns (Fingerprint of Things) in the images obtained with a regular camera.

1.Introduction

Recently an increasing trend in the various fields of the manufacturing industry is the application of IoT, with which the production history such as that of the processing and inspection of every manufactured product is collected and analysed as big data. The results are used to improve the quality control and production efficiency as well as supporting the design and processing accuracies. Unlike the original IoT with which a sensor was embedded in each individual product in order to obtain its data, collection of data from a product or part being manufactured necessitates the individual recognition to associate each piece of data measured by an external sensor or processing device. The individual recognition has usually been done by writing the serial number or attaching a barcode or RFID tag to each object. However, as these traditional individual recognition methods are not applicable to recently developed electronic components of extremely small size or to precision machines and materials that do not accept surface processing, it cannot be used to acquire individual data on the production history.

At NEC we have developed the Fingerprint of Things, this is a technology that is capable of the high-accuracy recognition of every individual piece of uniformly manufactured industrial products. This is done by applying image recognition of fine surface patterns (Fingerprint of Things) to the images of the things obtained by simply photographing them with a regular camera. This technology achieves individual recognition of various objects to which traditional ID tags are inapplicable and it thereby enables the use of IoT to record individual data of them. Below, we introduce the implementation of traceability, production innovation and product/component maintenance & inspection services that make use of the IoT by discussing individual component recognition in an example of the machine manufacturing industry.

2.IoT Based on Individual Recognition of Mass-produced Components

The traditional IoT used to handle the final product as a single “thing”, however, the final product actually consists of a huge number of components. In the manufacturing and maintenance supply chains, each individual piece of such components are also “products” that are “things” to which application of the IoT is desirable. For example, every aircraft is composed of a huge number of components. From tires and brakes to small items such as nuts and bolts used to retain the other components. Each of these components should be equipped with optimum performance and quality in order to avoid critical accidents. Even when a defect is found in only one of them, it is required to trace back the manufacturing history to identify the aircrafts using those components having the same problem and to recall them. In addition, it is also crucial for safe operations that such huge numbers of components are maintained and replaced routinely and optimally. In the field of maintenance, personnel inspect and replace a huge number of components that are mounted on aircraft including bolts and fasteners. They also manage and check the manufacturing history data.

If the IoT is applied to the management of the history of processing and work associated with such a huge number of components, it can contribute significantly to the achievements of both the efficiency and safety. Recently, the use of various IoT solutions have been expanding to identify individual things by attaching ID tags carrying barcodes or RFID and storing and analysing data such as the manufacturing history and usage situations. However, there are actually many things to which the IoT cannot be applied due to the impossibility of individual recognition. These include components that have sizes too small for attaching the tags, such as bolts and electronic components, as well as those that do not feature surface processing, such as precision machine components and materials. Even with components to which the tags can be attached there is also the problem of the very high cost required to attach the ID tags to the huge number of components (Fig. 1).

Individual recognition of such objects by applying the Fingerprint of Things, instead of by using tags, makes it possible to expand the various advances available via the IoT. For example, registering the Fingerprint of Things, images of each component on the component manufacturing line in a database and associating such data with the individual manufacturing history, specifications and inspection result data is performed as shown in Fig. 2. It thereby becomes possible to automate the checking of authenticity and specifications of each component as well as the recording and inspection of its usage circumstances in all of its shipping destinations and usage sites worldwide.

3.Fingerprint of Things Identification Technology

3.1 Fingerprint of Things

The Fingerprint of Things identification is a technology for implementing individual recognition and authentication. This is achieved by capturing images of the surface pattern specific to each individual piece of a product or component (Fingerprint of Things), which is produced spontaneously in the manufacturing process, and by then matching such images. The recognition of even very small individual components or of those that cannot allow process on the surface is possible by simply recording their images under magnification and optimum lighting conditions with a standard camera1), 2).

A general idea about industrial products is that there may be no individual difference between industrial products of identical specifications that are manufactured to avoid variances by using manufacturing equipment of identical specifications. However, when the surface image of a component is photographed under a specific lighting condition and it is magnified using a microscope, it is actually observed that the surface irregularities of each component has a slightly different pattern from those of the other components (Fig. 3). These differences are naturally not at a level that would affect the performance and quality of each product or component, but the presence of differences between individual pieces can still be recognized from such images.

Whether it is a machine component manufactured with high-accuracy cutting or a component made from the same mould, the “Fingerprint of Things” of each individual component still has unique features different from the others. This can be observed with any image obtained under optimum lens and lighting conditions using a regular camera such as that of a smartphone. In short, as do living things, industrial products also have “fingerprints” unique to each individual object that may be used for individual recognition.

Surveys of a large number of various industrial products and components carrying the same model numbers have found that each individual item has its own unique “Fingerprint of Things” on its surface (Fig. 4). Matching of the identicalness of such fingerprint patterns enables the individual recognition of each piece of a huge number of objects.

3.2 Photographing the Fingerprint of Things

Highly accurate individual recognition/authentication of mass-produced components requires the fingerprint of each individual piece to be photographed and registered in a database. Although the size and fineness of a fingerprint varies between things, high accuracy of recognition can be achieved by using optimum lens and lighting conditions so that the acquired image can clearly represent the unique features of each individual piece.

In this paper, we describe an example of the method of photographing a “fingerprint” from a metallic component (the method is variable depending on the properties of the component). Specifically, we individually selected the recognition of the bolts that are representative of the mass-produced components. We then developed a photographic method assuming the use of a mobile terminal that is easily distributed to the user, and prototyped a photographic device, aiming thus at the auto-collection of the Fingerprint of Things, images of individual bolts.

In order to photograph the fine profile irregularities of glossy surfaces such as metallic surfaces as fingerprints by using a regular camera we have developed a lighting system called FIBAR (Fingerprint Imaging by Binary Angular Reflection1). FIBAR is designed to capture the individual dissimilarities of object surfaces as stable images of Fingerprint of Things by enhancing local inclinations as shades.

Fig. 5 shows a FIBAR mechanism that utilizes ambient light without the need of a special lighting device that may be implemented via a smartphone or a low-cost mobile device such as an industrial tool, so that the user of the components can easily use it. Simply mounting a low-cost mechanism such as this on a mobile device or industrial tool enables photographing of the Fingerprint of Things and of their individual recognition by its use.

On the other hand, the manufacturing lines engaged in mass-production need a system that can acquire the images of a large quantity of components one after another. Fig. 6 shows a registration system for the Fingerprint of Things that automatically captures the fingerprints of a large quantity of bolts and builds an individual recognition database by using them. With this system, bolts in the specified pose (so that they face the camera from the front) are conveyed and photographed immediately below the camera sequentially from the right. To take pictures of the components flowing one after another over a short period, the FIBAR mechanism employs LED lighting and enhanced brightness contrast with identifiably high quality Fingerprint of Things images even when the camera’s shutter open time is short. Fig. 7 shows some examples of actually photographed Fingerprint of Things depicting the bolt surfaces.

3.3 Matching the Fingerprint of Things

The images of Fingerprint of Things can be matched using an image matching technique based on local feature amount and an algorithm for verifying the consistency of the geometric layout of feature points. First, the points where there are sharp brightness changes and the positions of which can be determined stably are defined as feature points. Next, the local brightness patterns around the feature points are extracted into the feature descriptor data. Then, from both of the images to be collated, feature points with minimum differences in feature descriptor are obtained as pairs. Finally, the geometric consistency of the overall arrangement of the feature points is verified so as to select only those pairs without inconsistency in their relative position relationships with other feature points.

Assuming that the total number of feature points extracted from the two collated images (the photographed image to be collated and the image of each individual item in the database) is Ntotoal and the number of the feature points pairs with correct geometrical layouts is ninlier, S, the score of collation of two images, can be calculated using the following formula.

S = ninlier / Ntotoal (1)

When the score of collation is higher than the specified threshold, it can be determined that the two images are of the same individual item.

When the Fingerprint of Things of the component in hand is matched with those of all individual items in the database, only the identical item achieve the high score above the threshold so that it can be identified. If there is an individual item not registered in the database, which means that it is a different or counterfeit component, it does not achieve a score above the threshold so it can be identified as an illegal item not covered by the database. Fig. 8 shows an example of collation of identical individual items and that of collation of different individual items.

4.Example of IoT Implementation Based on Fingerprint of Things

Fig. 9 shows an image of the use of IoT for preventing human errors in aircraft maintenance making use of identification of individual bolts based on the Fingerprint of Things identification. When each and every individual piece of the massively employed bolts can be identified with the Fingerprint of Things using a camera embedded in an industrial tool, it will contribute greatly to the elimination of human errors including component misuse and checking omissions3). In addition, such a system is also capable of automatically storing a large amount of routine inspection data in the cloud, including records of when who attached each bolt at which torque. It is expected that AI-based analyses of the big data automatically acquired as described above will bring about new kinds of knowledge such as the discovery of positions frequently causing trouble.

5.Conclusion

In the above, we described that the individual recognition based on the Fingerprint of Things of various objects that do not accept the attachment of traditional barcodes and tags can expand the applications of IoT to various scenarios, such as tracing and manufacturing to the inspection of component parts. The Fingerprint of Things is a key one that automates the identification of objects that do not accept tags and expands the IoT to include all things.

We intend to deploy this technology for application in a variety of new fields.

Reference

1) T. Takahashi, R. Ishiyama: FIBAR: Fingerprint Imaging by Binary Angular Reflection for Individual Identification of Metal Parts, Proceedings of the 5th International Conference on Emerging Security Technologies (EST-2014), pp.46–51, 2014

2) R. Ishiyama, et al.: Machinery Product Authentication by Matching Image Features of Micro-Bumps on Painted Surface, Journal of JSPE (The Japan Society for Precision Engineering), vol.82 (3), pp.251-258, 2016.3

3) Toru Takahashi, et al.: Intelli-Wrench: Smart Navigation Tool for Mechanical Assembly and Maintenance, Proceedings of the 24th Annual ACM International Conference on Multimedia (ACMMM), pp. 752-753, 2016.

Authors’ Profiles

ISHIYAMA Rui
Principal Researcher
Data Science Research Laboratories

TAKAHASHI Toru
Assistant Manager
Data Science Research Laboratories

KUDO Yuta
Data Science Research Laboratories

Related URL