Optical vibration sensing technology is the world’s first technology to utilize video to estimate the level of deterioration within critical public infrastructure, such as bridges. This technology precisely measures surface vibrations on the structure being analyzed and then estimates the degree of internal deterioration - i.e., cracks, flaking, and cavities - based on the characteristics of those vibrations. This means that it is now possible to analyze the interior deterioration of a massive structure from ground level with no need for scaffolding. As well as enabling early detection of deterioration and significantly reducing inspection costs, optical vibration sensing makes it possible to quantify the amount of internal deterioration irrespective of how skilled or experienced the inspectors are. Future iterations of this technology will deliver even more sophisticated capabilities as we continue to improve deterioration transition forecasting and automatic generation of maintenance plans.

1.Introduction

During the period when Japan enjoyed rapid economic growth, a huge number of public infrastructure projects were constructed throughout the country. Today, as these structures age and become increasingly dilapidated, there is a growing need for an effective and efficient system for maintenance and repair. However, the sheer size of many of these structures, such as bridges and tunnels, makes it extremely difficult for engineers to conduct close inspections. With accurate assessments critical to proper maintenance, the demand for technology that will make inspections more efficient has never been stronger.

NEC has developed optical vibration sensing technology that meets this challenge head on. By detecting and analyzing micro-vibrations in structures based on video camera footage, this technology makes it possible to easily identify points where internal deterioration - such as cracks, flaking, and cavities - is occurring.

This paper will provide an overview of NEC’s optical vibration sensing technology and discuss how the application of this technology can significantly improve infrastructure maintenance and inspection.

2.Issues in Infrastructure Maintenance

Many of the structures that support Japan’s transportation networks were built in the 1970s. With so many structures reaching their assumed life span of 50 years at almost the same time, the danger of collapse is very real. Reliable inspection, maintenance, and timely repairs are critical to forestalling this threat. Unfortunately, the number of people with the skills and experience required to handle this monumental task is actually declining year by year, making it more difficult to keep up with regular maintenance. On top of that, the financial difficulties of the Japanese government make it difficult to come up with the funds necessary to cover the costs of inspections and repairs. All of this has combined to create a worsening situation that has led many to fear that it will soon no longer be possible to ensure the safety of the nation’s infrastructure - at least, as long as conventional inspection and repair methods continue to be used.

When inspecting major infrastructure, especially concrete structures, it is of vital importance to accurately diagnose any internal deterioration such as cracks, flaking, and cavities, as these conditions significantly impact strength and safety. Currently, inspectors check for internal deterioration by hammering the surface and analyzing the resulting echoes. This technique not only requires experienced personnel, it requires that scaffolding be erected so that inspectors can examine the structure directly. This scaffolding alone is estimated to account for about half the cost of inspection. Eliminating the need for scaffolding, then, would result in tremendous savings.

Another problem with the current inspection regime is that diagnostic results vary depending on the experience and perception of the inspector. In Japan, where many skilled workers are aging out of the workforce, demand is now growing for a diagnostic technique that achieves constant results regardless of the worker’s experience level.

3.NEC’s Optical Vibration Sensing Technology Facilitates Remote Estimation of Internal Deterioration

NEC’s answer to these problems is a remarkable new technology called optical vibration sensing. Developed drawing on the company’s world-class expertise in video and image processing, this technology makes it possible to estimate deterioration inside a structure from a remote location simply by capturing images shot with a conventional visible light video camera. No special wavelengths or laser irradiation are required.

Data obtained from a visible light camera normally pertains only to the surface of the structure. The key to estimating internal deterioration is to use the “vibration” data obtained from the video images.

One of the major factors contributing to the deterioration of large structures such as bridges is the application of external force, particularly vehicle traffic, which can lead to deflection, distortion, and so on. It is well understood that the vibrations that occur on the structure’s surface when an external force is applied are directly related to the state of internal deterioration. The standard technique for diagnosing whether or not there is internal deterioration is to hammer the structure and listen to the reverberations in order to identify response vibrations from the surface. It is also known that cracks on the bottom surface of floor slabs of a bridge open and close in the order of 10 µm when a heavy vehicle passes over the bridge.

Without taking account of these dynamics, conventional image analysis methods perform diagnosis by using only one image for one point. This not only makes it difficult to differentiate between stain lines and fine cracks on the surface, it also makes it impossible to determine whether a crack is only on the surface or has penetrated the structure, e.g., flaking.

However, advances in imaging technology have opened up new possibilities. The speed of the image pickups used in the latest video cameras has increased significantly as has the speed of semiconductors. Most people are not aware that some of the high-speed cameras used today for product inspection in factories boast shooting speeds as high as10,000 frames per second. With such a camera, it is possible to observe in detail the surface behavior of a structure. NEC’s optical vibration sensing technology exploits this high-speed shooting capability to gather dynamic information about the structure and estimate the degree of internal deterioration.

Fig. 1 shows the processing flow of the optical vibration sensing technology. A video camera is used to capture images of a structure, such as a bridge, while it is subject to traffic excitation. Vibration data is obtained from the captured images and analyzed. Interior deterioration is estimated based on vibration data patterns.

3.1 Using Camera Footage to Measure Surface Vibrations on a Structure

The first step in the process of optical vibration sensing is surface vibration measurement (Fig. 2) which is based on tracking pixel patterns in the camera footage. Essentially, this involves precisely determining where the pixel pattern at each coordinate in one video frame has moved (been displaced) within another video frame. By taking advantage of NEC’s expertise in pixel pattern search developed for use in super-resolution technology and video image compression encoding technology, we have achieved high-speed, high-precision movement search on the order of 0.01 pixels.

Displacement direction and amount at each time and in each coordinate are then graphed chronologically to obtain vibration data. When a 2 x 2-m area is shot with a 2,000 x 2,000-pixel camera, each pixel is 1-mm square, making it possible to capture displacement on the order of 10-µm displacement, which is 1/100 of the pixel size. Vibration data is collected for 1 to 2 seconds for each location. This data is required for processing in the subsequent steps.

In principle, because this processing tracks minute patterns on the surface of the structure being inspected, the surface of the object should have patterns. This is generally not a problem, as the one of the materials most commonly used in these structures is concrete. However, it is not effective with plain, transparent, and reflective material.

3.2 Estimation of Internal Deterioration Based on Surface Vibration Patterns

The second step in the process is estimating the amount of internal deterioration based on the surface displacement (patterns) that emerges when the data is viewed chronologically.

To show how this works, we will describe a simple experiment using a soft material. In this experiment, we used a pressurizer to apply pressure to a styrofoam beam (resembling a bridge girder) from above to simulate traffic vibration. Fig. 3 shows chromatic shading visualization of the degree of displacement obtained from the analysis results based on footage of the bottom surface. Specimens that contained internal deterioration such as cracks, flaking, and cavities showed different patterns of surface displacement from those that were free of deterioration. We can infer that the difference in displacement patterns occurred due to changes in surface stress conditions as a result of internal deterioration (Fig. 4).

In a specimen with no deterioration, the surface displacement when load is applied is represented as a smooth gradation indicating that the pressure accompanying the load is uniformly distributed across the specimen’s surface.

On the other hand, when a specimen has cracks or flaking, the color density becomes discontinuous along the borders of the cracks or flaking. What this means is that the damaged area - a crack, for example - opens and closes when a load is applied, making the direction and size of its displacement different from any displacements nearby. Moreover, when cracks have penetrated the interior, the density of colors immediately below the flaking becomes constant (because force is not applied directly).

When a specimen contains a cavity, the displacement created on the bottom surface right below the cavity appears smaller than other displacements in the vicinity. This phenomenon occurs because the cavity distributes the loading pressure from above as if an air cushion were applied, reducing stress just below the cavity.

When these different displacement patterns are input into analytic processing, the amount and type of internal deterioration (cracks, flaking, and cavities) can be estimated (Fig. 5).

It is important to point out that the camera itself vibrates in actual usage, but this technology can differentiate between discontinuous vibration caused by deterioration and self-vibration. This is because all the pixels in the image move in the same direction when the camera vibrates.

Another important point to be noted is that this technology does not capture long-term deformation over extended periods of time. Rather, it performs estimation by using only observation results from a limited time period (1 to 2 seconds). It therefore makes it unnecessary to observe the same location for years — which is the case with many of the current available methodologies.

4.Reliable and Efficient Infrastructure Maintenance

The positioning of the optical vibration sensing technology is shown in Fig. 6. When applied to infrastructure inspection, this technology makes it possible to: (1) lower the cost and (2) detect internal deterioration.

At NEC, our goal is to improve overall inspection efficiency by making the best possible use of the data that can be easily collected using our technology. For instance, internal deterioration data will be processed with higher-level machine learning, allowing us to predict future deterioration transitions and to optimize repair plans, while specifying repair schedules and locations.

Generally speaking, machine learning requires large-volume learning data in order to achieve sufficient accuracy. In conventional infrastructure maintenance, there are limits to how much inspection accuracy can be improved by machine learning because it is simply not possible to gather sufficient data. The need to allocate resources for personnel, the time required, and the cost of inspections themselves all conspire to frustrate data acquisition. Furthermore, because they are based on human perceptions, many archived inspection diagnosis results are insufficiently quantitative and objective.

NEC’s optical vibration sensing technology provides a powerful solution that can overcome most of these problems, especially cost, accuracy, quantitativeness, and so on. Reinforced by AI technology, optical vibration sensing technology will make it possible to support a comprehensive next-generation maintenance environment including planning strategies and policies.

Going forward, we plan to expand the capabilities and versatility of this system by integrating other sensing technologies with the optical vibration sensing technology, as well as by incorporating additional data (designing information, traffic volume, budget, etc.) to create a comprehensive system for managing infrastructure maintenance and repair (Fig. 7).

5.Conclusion

In this paper, we have introduced NEC’s optical vibration sensing technology and explained how this technology can be applied to the maintenance of public infrastructure, providing a more reliable, more efficient, and more cost-effective means of identifying internal structural deterioration. Currently in the verification stage, this technology is now being tested on large-scale concrete models and actual structures in cooperation with various organizations and operators. The advantages of this technology cannot be overstated, and we believe it is imperative to put it into practical use as soon as possible in order to ensure public safety and address the potential problems posed by aging infrastructure.

Authors’ Profiles

TAKADA Jun
Principal Researcher
Value Co-creation Center