June 30, 2020

NEC has developed a social distancing detection technology that allows detecting and visualizing in real-time whether people are keeping a sufficient distance apart in gathering sites such as stores and public facilities including train stations and airports. In this interview, the researchers talked about this technology which can be introduced in just a few minutes using existing cameras.

Visualization of risk in just a few minutes using videos from existing cameras

― What kind of technology is the social distancing detection technology?

Nishimura: This technology analyzes camera videos and allows the user to detect whether people are keeping a suitable distance apart (social distancing). It can detect human bodies in the videos and determine the degree of crowdedness by analyzing whether people are in contact within a circle of 1 m radius (radius can be set freely by cm). We believe that this technology can contribute to preventing the spread of infections.
While now in Japan (interview date: June 11, 2020), there is the view that COVID-19 infections have subsided (^*1), there remains the risk of a second and third wave. Measures such as social distancing continue to be important.
This technology is the first new technology launched by NEC to prevent the spread of COVID-19 infections. We started development at the beginning of May and aimed to commercialize it as quickly as possible, with the thought of whether we can make a contribution by utilizing our knowledge on video analysis which our team has been working on.

Nakano: The most revolutionary point is that it can use existing cameras. This means that the commonly used cameras can be used without modification. Although the existing technologies are able to measure distance between persons in camera videos, careful preparation is required. For example, it may be necessary to physically go to the site where the cameras are installed, place markers in various locations within the field of view, and measure the distance between them. Because video includes a perspective (depth), it is necessary to perform on-site measurement of actual person sizes and distances for analysis. However, this involves personnel and expense, and several days of preparation are required before system operation can be started. For highly public locations such as intersections, introduction of this kind of system involves tremendous difficulties due to the need to stop traffic for a time. While there are also high-performance cameras that are able to measure distance, they are limited by the narrow view angles that can be captured and measured.
In contrast, our technology does not require any on-site measurement. After existing videos are uploaded to the designated server, analysis of the videos is completed in just a few minutes and it is possible to make highly accurate social distancing detections. In a demonstration, we analyzed existing videos of a square in Budapest and Shibuya Scramble Crossing. These are locations where on-site measurement is not possible, and the results proved that proper analysis was possible even in wide-angle video covering a large area.

Nishimura: Another technology that can help prevent infection is the one that measures the distance between smartphone devices. However, this technology is more effective for investigating the paths of infection. Our technology focuses on measuring the distance between people, and makes it possible to accurately identify social distancing from videos. There are large advantages to being able to determine the degree of crowdedness according to the location and time of day, and to visualize and quantify the results.

― In what kind of situations do you think this technology will be used?

Nishimura: We expect that it will be used at places where people frequently gather such as public facilities including airports and train stations, street corners, stores, and shopping malls. We think that there will be two primary ways that this technology can be used. The first is as a method of providing information to visitors. For example, we can inform and request visitors to avoid high-risk times when using the facility by displaying on digital signage time periods and areas where a high level of crowding risk was identified with this technology. The analyzed video can also be displayed so that it can be seen by visitors. Because it allows suitable social distancing to be visualized, we believe it can contribute to preventing the spread of infections.
The other way of using this technology is for risk management and ensuring safety of facility staff. Because it allows high-risk time periods and areas to be identified, it enables improvements and measures such as avoiding having staff work for prolonged periods in high-risk locations.

Rapid development by bringing together NEC technical assets and organizational strengths

― How did you develop this technology?

Nakano: The basis was a technology for estimating the camera position in three-dimensions from 2D videos. This is what I had been studying since last year. This technology uses video analysis to estimate where the shooting camera is located in real space. We were developing it as a technology for measuring sizes and distances of objects in videos. Originally, this technology was intended for use with objects.
However, as our team was thinking about what we could do in response to the COVID-19 pandemic, we came upon the idea of combining this technology with human body detection technology and applying it to measuring distances between persons. When we actually challenged ourselves to achieve this, we succeeded in estimating the camera position and three-dimensional distances using reverse calculations based on comprehensive analysis of the sizes of moving people and other characteristics in the video data, and were able to successfully release this technology.
One of our team members is developing a high-accuracy engine for human body detection technology in the same Biometrics Research Laboratories, and we were able to utilize that engine.

Nishimura: NEC has assets that it has developed over many years in the computing and network fields. This new technology was not suddenly output starting from zero. In addition to having a close researcher who was developing a human body detection engine, we were also able to utilize detailed operational knowledge and field experience as a result of advanced “crowd behavior analysis” technologies that analyze the density and conditions of crowds. We were also greatly aided by a precise understanding of customer needs and accurate feedback from business divisions based on the relationships they have built with customers.
With the abundant technology platforms and knowledge that we have inherited, we were able to bring these assets together in this research to create this new technology. Assets of this kind are one of our large strengths here at NEC.

Towards new protection technologies that support our distanced society

― You created this new technology quite quickly, didn’t you?

Nishimura: Yes we did. We started discussing what we could do to combat the spread of COVID-19 infections in April, and it was after the consecutive holidays in May that we were able to present the idea for this technology. We moved fast, and it was just around a month before we issued the press release. In fact, this is the first time I have met directly with Mr. Nakano since this project was started (laughs). We had been communicating with each other remotely.

Nakano: Yes, that’s right. We accessed the company PCs remotely, and did all of the programming from home. Our brainstorming was also done online and involved around 20 people, including members from business divisions and patent office. Maybe our information sharing went so smoothly because of online.
For my part, I was grateful that Mr. Nishimura took on the behind-the-scenes work so I could concentrate on the technology side.

Nishimura: I was amazed that we were able to give shape so quickly to something that did not exist at all just a month earlier. We coordinated bottom-up with business divisions on a variety of matters, and top management made proper decisions on a top-down basis. The two sides fit together perfectly. I think that this was the result of the combined efforts of everyone who participated in the project.

Nakano: We have submitted a research paper on camera position estimation technique from which this technology was developed, and it is now being reviewed by the 25th International Conference on Pattern Recognition (ICPR 2020). (After the interview, it was officially accepted. The presentation will be done in January 2021.) We want to actively promote our technologies to the world, including applications of this social distancing detection technology.

― What kind of future development are you planning?

Nishimura: The mission of our research team is to contribute to the security, safety, and greater efficiency of society through video analysis. In that sense, this technology can be seen as an application which is part of this mission.
In the post-coronavirus society and our current battle with the virus, the value of touchless and remote technologies are likely to take on greater importance. Unmanned shops and autonomous driving lie farther down this same path. If these technologies expand into our cities, we will see greater labor savings and automation, and there will be growing cases when there are no persons tasked with managing and monitoring the surrounding conditions. We believe that this “protection” using video analysis will play a large role in these developments. I hope that our video analysis technologies and solutions will contribute as a means of creating safety and security that is well-suited for our new social situation.

Go back to Featured Technologies