One of the objectives of edge computing is to provide the advantages of cloud computing in AI and big data processing while minimizing the use of IT resources in the real world. This paper introduces five cases of edge computing solutions offered by NEC. Two of the examples, the walkthrough face recognition system and the human behavior analysis service, perform image processing such as face recognition at the edge layer, then perform actuation at local or behavior analyses on the cloud. This paper also describes cases like use cases of beacons, collecting and visualizing temperature data via sensor with the edge computing and NEC’s activities to implement the solution with IoT smell/odor sensor.

1.Introduction

Edge computing enables optimization of the quantity of data sent to the cloud system and its real-time actuation in the field. It does this not only by collecting sensor data in the edge layer installed in the field but also by processing and analyzing such data. This procedure can avoid a concentration of processing operations in the cloud layer so that the IT resources of the cloud layer can be optimized and their availability improved.

In this paper, we cite examples of NEC’s edge computing solutions and specifically we introduce the role of edge computing in the field and discuss the functions that it provides.

2.Solutions Based on Face Recognition

2.1 Walkthrough Face Recognition System

At NEC we possess a face recognition technology that is ranked No. 1 in the world. This technology enables us to deploy products such as NeoFace to deal with businesses based on face recognition procedures that are expected to expand in the future. We are developing walkthrough face recognition systems as the themed solution products to be connected with entrance/exit gate and door.

Although ordinary security gates authenticate individuals using IC cards the security level can easily drop due to any loss or lending of the IC cards. Such an eventuality necessitates authentications based on biometric information linked to each individual. However, for conventional biometrics like fingerprints and vein patterns, a special device is required in order to obtain relevant biometric information and a confirming action is required via the readout device at the time of authentication.

On the other hand, face recognition can obtain biometric information easily from available photographed images. Also at the time of authentication, no special action is required if the face can be captured by an installed camera. All the authenticated person has to do is to walk through a certain area.

The walkthrough face recognition entails: 1) defining the point at which the authentication target is captured 2) completing the authentication in a short walkthrough period 3) taking appropriate procedure is required. These operations are implemented by a new face recognition technique called “pre-enrolling” (Fig. 1). Pre-enrolling performs authentication using the images captured from the entry of the target into the set area by matching. It is expected to achieve a high authentication accuracy even from natural motions and contribute to cost reduction due to resource distribution for authentication.

We exhibited a demonstration machine incorporating this technique at RetailTech Japan 2017. Walkthrough face recognition was experienced by more than 1,400 persons over four days and the system was well approved by them (Photo).

The walkthrough face recognition system is scheduled to be released during FY2017.

We are planning to adopt the device as the edge computing for the face recognition cloud service so that it can be used over a wide range of applications and customer needs.

We will develop a wide range of face recognition solutions to improve customer convenience while conforming appropriately to any legal restrictions or privacy issues.

3.Human Behavior Analysis Service Using Image Analysis Technology

Hitherto, acquisition of purchase activity information of visitors to retail stores has been possible only from the purchase records in POS data or via visual observation. As a result, the analysis of purchaser behavior prior to purchase and of non-purchasers who leave the store without making a purchase have been issues for the persons in charge of marketing in the stores. We have resolved these issues by installing cameras in the store to visualize the in-store activities of visitors by using image analysis technologies that detects people, monitors their movements and extracts their flow lines.

First, the images from the cameras installed in the store are collected by computer (edge) in the store. The analysis engine installed in the computer recognizes the moving object area, matches images of the head parts and extracts persons from the camera images, predicts their movements and tracks them. The analysis engine processes these operations in the temporary memory (RAM) of the computer and saves only the coordinate data of the extracted persons and dispose the captured camera images or the data created by the analysis engine during processing. One of the features of this system is that the data leading to the identification of individuals is not retained.

The generated human coordinate data of each camera are collected in the cloud system and the coordinate data from multiple cameras is linked in the cloud system in order to obtain the flow lines of persons. These are then compiled into various analysis results to be presented as the “heat map representing an in-store dwell time of customers,” the “number of humans passing through the in-store aisles per time zone” and the “conversions representing the ratios of shelf-front passage time.” As all of this information is located in the cloud, the person in charge of in-store marketing can refer the behaviors of store visitors via the web, from any place, whether it is in the office or outside the store.

This system is additionally cooperated with the FieldAnalyst system that estimates gender and age groups automatically so that the results of the extraction from the store camera images of the “number of visitors and gender/age groups per time zone” are also collected in the cloud for browsing (Fig. 2).

The obtained information allows the person in charge of marketing to analyze the behaviors of visitors continuously and quantitatively and facilitates planning of actions based on the acquired data, such as for store layout decisions, as well as verification of the effects of actions taken. In addition, when the information is combined with the purchase records in the separately acquired POS data, the non-purchasing behavior of visitors can be analyzed in addition to the purchasing behavior.

Fig. 3 shows an example of an analysis display of this system.

4.Using the NEC Mobile Backend Platform

Due to the increased business use of the cloud computer and smart devices, the present ICT system developments often presuppose “cloud first” or “mobile first,” but customers very often request a slow start to their system introduction. We have therefore developed the NEC Mobile Backend Platform software that turns the standard functions for the cloud computer and smart device linkages into common parts as an early provision for customers (Fig. 4). Also applicable to the field of IoT, this software stores various information collected from sensors via an edge device in a DB and provides access control per data to provide information only for qualified persons. The information can be browsed not only from an app or browser but may also be sent to a smartphone or tablet terminal using the push notification function. Each customer is also allowed to define a unique API and to perform user-defined processing in a server. Two of the cases in which smartphone or tablet terminals are used as edge devices are introduced in the following subsections.

4.1 A Case Using Wearable Devices

Wearable devices such as smartwatches are gradually gaining popularity with consumers and enterprise usage is also expected to increase.

For example, to manage the working conditions of employees in fields such as construction and at facility inspection sites, vital information of each worker wearing a smartwatch can be collected and sent to the cloud computer by using an incorporated biosensor. Based on this information, the field manager can check the situation of each worker by means of the push notification.

The burden of worker response can be reduced by using the voice input function of the smartwatch or by setting a simple input interface to provide options for selection (Fig. 5).

4.2 A Case of Human/Thing Location Management Using Devices Such as Beacons

The need to improve work efficiency for productivity improvements in factories and at construction sites are now higher than ever.

In order to improve efficiency it is first necessary to determine and visualize the flow lines of workers and vehicles on site. However, if these are surveyed manually by human labor in the traditional way the time taken to do so would adversely affect efficiency and the use of IoT technology enables efficient visualization.

The location can be determined by collecting the movement information on sensor devices, by then mapping it via location information and finally by displaying it and recording flow lines on the map.

To collect the movement information, it is necessary to select devices according to the current environment. In the cases outdoors, GPS is generally used but QZS has also been used recently. In the case of indoors, some devices are available, including the UWB (Ultra-Wide Band), Wi-Fi and acoustic wave devices but the beacon is suitable in many cases because of its affordability at a few thousands of yen per beacon.

The information output from beacons can be received on smartphones or on tablet terminals with BLE (Bluetooth Low Energy) compatibility (Fig. 6). The beacons transmit their IDs at constant intervals so it is necessary to complete mapping of the IDs and locations on the indoor map in advance. The data received by the smartphones and terminals is saved in the NEC Mobile Backend Platform and is visualized by converting it into actual location information and displaying it on the map (Fig. 7).

When the push notification function of the NEC Mobile Backend Platform is used, alarm notification can be sent to a smartphone or a smartwatch worn by someone who approaches a restricted area.

In the case of beacons, they use 2.4 GHz band radio waves, so it is necessary to consider interference from adjacent beacons and other equipment.

In the present case, the smartphones functioning as the edge devices execute anti-interference processing in order to enable location determination with higher accuracy.

5.IoT Data Visualization Solution

The data stored in large amounts of equipment without communication network connectivity in factories and stores as well as in cameras and temperature/humidity sensors can be visualized by connecting them to edge terminals. These must be compatible with multiple wired or wireless interfaces such as edge gateways and linked to the cloud system. Visualization of previously unavailable data such as the monitored camera images and machine tool working status in the cloud system makes it possible to identify and analyze the current situation in real time. The results can be utilized in various customer services including failure prediction and remote management functions.

In the following subsections, we introduce cases of visualization including kitchen temperature visualization using an edge gateway and the visualization of factory vibration using vibration sensors.

5.1 Kitchen Temperature/Humidity Monitoring for Food Security and Safety

As seen from the promotion of HACCP by the Japanese Ministry of Health, Labor and Welfare, we find that the eating-out industry is now urged to improve food security and safety. As one of the suggested measures, we propose a temperature management solution that records the temperatures in the kitchen refrigerators and dining halls and then outputs the data measured with the temperature sensors as business form data (Fig. 8).

If a temperature warning occurs, the proposed solution is notified to the mail address that has been registered in advance on the service platform in the cloud system, so that employees can be freed from their temperature checking work.

Automation of temperature management tasks that have previously been done manually by employees can improve the overall work efficiency and the quality of the temperature measurement/management. Such procedures will also help to protect the food quality against failures of the kitchen equipment or forgetting to close the refrigerator door.

In the future, we are planning to implement AI engines in the cloud computer and at edge gateways with the aim of providing enhanced values from visualizations and data analyses.

5.2 Factory Vibration Visualization Solution

In factories such as steel plants, it is now considered to be essential to inspect manufacturing equipment in order to ensure the satisfactory implementation of the production plan and of production quality. We offer a solution that visualizes the vibration related situations of a factory by sensing the vibrations of facilities and equipment in order to check a facility anomaly status. In the future, we intend to also offer a solution that detects signs of irregularities by analyzing the vibration data with an AI engine implemented in the cloud computer (Fig. 9).

6.Solutions Using Olfactory IoT

6.1 Outline of Olfactory Sensor

It is said that the number of odor molecules is as large as hundreds of thousands, and that each “odor” is formed by blending tens to hundreds of these molecules in a specific concentration ratio. Consequently, the number of combinations is almost limitless and their complexity has made the olfactory sensor delayed most among sensors around the human senses. Provision of a sensor that can easily measure and identify odors as an IoT solution is expected to contribute to various fields including that of food management, environmental measurements and safety checks as well as medical and health wellness, such as breath diagnosis (Fig. 10).

6.2 MSS and MSS Alliance

The MSS (Membrane-type Surface stress Sensor) is a versatile, ultra-small, super-high-sensitive sensor element jointly developed in 2011 by NIMS (National Institute for Materials Science) MANA Independent Scientist Genki Yoshikawa, late Dr. Heinrich Rohrer, and École polytechnique fédérale de Lausanne (EPFL).The essential requirements for practical implementation of the MSS include optimization and precise evaluation/calibration of the sensor chips. Aiming thereby at mass-production as well as at the development of a standard module that integrates them with an analysis system. The MSS Alliance has been established to promote the development of these key technological elements.

At the MSS Alliance, NEC is in charge of flavors/odors analysis using the heterogeneous mixture learning technology, which is its original machine learning technology. It also manages the construction of an IoT platform that contains an edge gateway.

6.3 NEC’s Determination Engine and Analysis Platform

NEC uses the heterogeneous mixture learning technology developed independently by NEC for flavors/odors analysis. We employ a module incorporating MSS to acquire the waveform data of the adsorption and desorption of odor molecules on receptor layer. We also apply the heterogeneous mixture learning technology to waveform data, learning so that the feature selection that are effective for determination vary depending on the measuring conditions. Moreover, the receptor layer suitable for determinations are defined when the measuring conditions of the determination target are decided (e.g.: a certain receptor layer is to be used to determine certain gases at a temperature of 20°C or higher) (Fig. 11)

At nanotech 2017 held in February 2017, we demonstrated the results of verifications for estimating the hardness (a ripening index) of pears from their odor.

6.4 Edge Computing in Olfactory IoT

At NEC, we are planning to build service that collects the odor data acquired with MSS using an edge computing and stores it in the cloud system for future use. We are also planning to provide the flavor/odor analysis via this service, by using the heterogeneous mixture learning technology described above. At the same time, in addition to using the cloud system, we will run the analysis engine on the edge computing in order to improve the response time of the odor determinations (Fig. 12).

The features of MSS include its compact size and light weight. To make full use of these features, we aim to adapt the edge computing system for smartphones as well as stationary type devices. We expect that this will expand its range of applications, including offering a handy odor determination solution.

Authors’ Profiles

SAITO Kenji
Assistant Manager
IoT Platform Development Division

MIKAMI Akiko
Manager
IoT Platform Development Division

ARIGA Kenichi
Manager
IoT Platform Development Division

YASUTAKE Hiroshi
Manager
IoT Platform Development Division

KIMURA Shigeo
Manager
IoT Platform Development Division

HANE Hidetaka
Manager
IoT Platform Development Division