The use of robots in various settings to support human lifestyles has recently been gathering momentum.

In Japan, the reduction of workforces brought about by the falling birth rate and the increase in the aging population is attracting attention to the use of service robots in the fields of customer reception and caregiving.

This paper proposes methods of utilizing service robots based on NEC’s authentication technologies, such as that of face authentication. In addition, advanced customer communications that can be implemented based on linkages with the authentication technologies are also introduced.

1.Introduction

The greater part of the current Japanese robot market is occupied by industrial robots for use in factories and fabrication plants. Moreover, the market forecast of the Japanese Ministry of Economy, Trade and Industry and NEDO (New Energy and Industrial Technology Development Organization) is forecasting further expansion of the service robot market. It estimates that by 2020 its market scale will grow by 1 trillion yen, which is equivalent to that of the industrial robot market. Service robots are those that perform useful tasks including human reception, security and welfare services in supporting human lifestyles. The scope of service robots has recently been expanding because of the shrinking labor force that is a result of the falling birth rate and the increasing aging population.

A recent trend is a mechanism to support the automation of operations called the software robot or RPA (Robotic Process Automation) that is being introduced more widely than previously. The authors of this paper are keeping an eye on such trends but intend to focus on the robots accompanying the hardware as the main subject of this paper.

With such trends in focus we initiated an NEC in-house taskforce for the study and promotion of the usage of service robots in FY2016. Our R&D has been focused on linkages between the service robots of human-shape that are used in the reception services and in the care services and the “NEC the WISE” that implements NEC’s highly accurate AI technologies. In particular we are emphasising the importance of the authentication technologies.

Linkage of NEC’s authentication technologies such as the face recognition with the service robots makes possible advanced customer communications that are not possible with the robots alone. We believe that this strategy will allow us to create previously unavailable services and standards, including customer networks.

2.Linkages between Service Robots and External Services

In this paper, we introduce the linkages of service robots with external services as a means of giving advanced functionality to the robots. To utilize service robots effectively, it is critical to link them with external services via the network as well as to use the functions already built into them. Linkages with external services make it possible to enhance both quality and quantity of the services provided by the robots.

The general method of linkage between a robot and an external service is to connect to the API (Application Programming Interface) provided by the external service and to use its functions (Fig. 1).

The API is an interface used for the linkage of functions between application software components.

Because the APIs can be used only as needed at a required timing, it is easy to combine several APIs into a single service or to incorporate an API in an existing service.

Our taskforce has carried out the connection verification between NEC’s authentication technologies and robots by producing a pilot API for each of the authentication technologies to be verified. We have also developed robot applications for linking robots and APIs and have completed the actual running tests for various kinds of robots.

In the following sections, we will introduce the kinds of customer communications implemented by the linkages of NEC’s authentication technologies and robots and check the values of the advances thus achieved.

All or part of the various authentication technologies described below are the results of pilot productions by our taskforce. However this does not mean that NEC promises to provide all of them in the future.

2.1 Linkage via Face Recognition Technology

This section describes the linkage between NeoFace, NEC’s face recognition engine, and robots. The face recognition technology enables identification of a person from the image of a human face.

At NEC, we started R&D into the face recognition technology in 1989. Our technology is highly evaluated as witnessed by the top rankings for four consecutive times in the Face Recognition Vendor Tests of the U.S. National Institute of Standards and Technology (NIST) ¹⁾.

(1) Example of a face recognition linkage application at customer reception

A robot can identify each person visiting the reception of a financial institution based on face images so that each individual may be accepted appropriately (Fig. 2).

In this application, the robot shoots a face image and sends it to the API of the face recognition engine. The face recognition engine collates the face information with the database to see if the face image has already been registered. The collation is not performed using the image itself but uses the face features extracted from the image. Such a set of features extracted in this way is called the “feature quantity”.

When a face is found in the database, the robot acquires the customer information from the CRM (Customer Relationship Management) by using the ID tied to the face. The CRM has a list of the goods available for purchase. The list is developed by using NEC’s AI technologies and is prepared for each customer. The robot can then use this information in the customer reception.

The robot can also use its camera to register a face image immediately after shooting it.

(2) Values provided by face recognition linkage

When the robot has succeeded in identifying the customer, personalized recommendations are possible as described above. If the customer is not registered in the CRM, promotion of face registration can lead to the acquisition of new sales opportunities (channels).

Visitor reception by robot is expected to lower the psychological hesitation level of customers worrying about “would I be forced to buy something?” This is one of the advantages of the customer reception by robot compared to human effort.

2.2 Linkages between Image Recognition Technology and Robots

This section deals with the linkage of the GAZIRU, NEC’s image recognition service, with robots. Image recognition is a technology that links a recognition target with a camera and identifies the image on which it is based.

The GAZIRU service can recognize video displayed on a TV screen or digital signage as well as still images in magazines and brochures. In addition to such 2D materials, it can also recognize industrial products with defined shapes, such as automobiles, as well as natural objects with undefined shapes such as flowers and cooked dishes (Fig. 3).

Individual products have unique patterns called object fingerprints, just like the human fingerprints. Identifying object fingerprints enables the traceability management of objects for use in regularity judgements and in the management of specific goods.

The core technologies of GAZIRU have achieved excellent results in important trade contests worldwide. In Japan, the object fingerprint technology received the Economy, Trade and Industry Minister Award of the Grand Prix of Advanced Technology in 2016.

(1) Example of an image recognition linkage application

When a robot is linked with GAZIRU, passing a product brochure above the robot’s camera enables the robot to provide an explanation of the product (Fig. 4).

The robot sends the image shot with the camera to GAZIRU, which collates the image with the objects pre-registered in the database. As in the case of face recognition, the collation does not use the image itself but uses the “feature quantity” extracted from the image so that high-speed processing becomes possible.

Collation with the goods information database enables the acquisition of detailed information on each product for use at customer reception.

(2) Value provided by image recognition linkages

As seen here, a robot can provide explanations to customers by identifying real world objects and by adding information dynamically to the results. The information refers dynamically to information not given in the brochures, such as the price and inventory count.

Customers can benefit from the up-to-date information given by the robot in their selection of goods.

2.3 Linkages between Speaker Verification Technology and the Robot

NEC’s speaker verification technology performs collation between a preregistered voice database and a recorded non-programmed voice in order to identify the speaker with high accuracy.

(1) Example of a speaker verification linkage application

When a robot is linked with the speaker verification, the voice recorded with a microphone is sent to the API of the speaker verification engine to identify the voice of the person speaking.

Such a voice-based personal identification is usable in scenarios where authentication methods such as face recognition are unavailable, e.g., in a poor camera installation environment. As words different to those used in recording can also be matched, collation is possible in the course of natural conversations.

(2) Value provided by speaker verification linkages

The capability of voice-based personal recognition enables detection of a VIP guest or a complainer without letting them know that they are identified. In this way, a VIP guest can be treated specially by eliminating unnecessary procedures so that customer satisfaction may be expected to improve. On the other hand, a complainer can be handled by an experienced attendant in order to forestall a problem before it could occur.

Compared to other authentication technologies, we believe that speaker verification is an effective technology for dealing with such special circumstances.

3.Issues and Countermeasures before Extended Deployment

As described above, the combinations of NEC’s various authentication technologies and robotics can in many cases provide new values;

1) recognizing a person and recommending optimum choice of goods;

2) recognizing an object and offering useful information to support goods selection;

3) improving service quality and preventing problems in advance without the user being aware of it.

On the other hand, from consideration of extended deployment of robots, linkages with external services may present some issues.

(1) Lead time for authentication processing

For example, let us assume a case in which the face recognition described in section 2.1 is used as an external service.

Highly accurate authentication requires a highly stable image that shows a face clearly. To fulfil this requirement, the robot shoots the user’s face successively and sends the images repeatedly to the external service until a face image is obtained at a clarity level that permits collation. Although fast processing with less delay is required in order not to let the user be aware of the lead time, the time taken for the transfer of images cannot be ignored.

(2) Lead time reduction method

It is first required to prepare a computer that is connected to the robot via a low-delay, high-speed local network such as Wi-Fi. Next, it is also necessary to devise a method for processing the feature quantity extraction on such a computer (Fig. 5).

The feature quantity data volume is a few kB, which is only around 1/1000th of the camera image data. Sending only small volume of feature quantity data to an external service can significantly reduce the lead time for transfer processing and also increase the overall speed of the processing.

Such pre-processing can also be implemented by the robot alone.

However far the universalization of robot hardware and software is advanced, it is nevertheless a fact that many constraints will remain. Consequently, we at NEC believe that the optimum procedure should be to perform complex processing externally to the robot.

Furthermore, we consider the robot to be a device suitable for providing the following two functions;

1) Input to a camera or microphone, etc.;

2) Output of motion, LED or audio speakers.

Another key point that we should note in linking an external service and a robot is to ensure its security. Measures suitable for function enhancement would be the encryption of communication paths, distributed placement of data and impersonation prevention.

Our team is tackling R&D enhancement on a daily basis in order to enable these solutions to take on concrete forms.

4.Conclusion

There are innumerable circumstances in which service robots may be applied as an alternative work force for dealing with the anticipated reduction of labor input that will face Japan in the future.

For example, many enterprises and local government organizations have already started significant programs for supporting watch-over and recreation services at the assisted living service facilities for seniors.

As international big events are scheduled to be held in Japan in the near future, it is also expected that multilingual services to support inbound tourists will by then be even more necessary than previously.

In order to meet these needs promptly, it will not only be necessary to make full use of the technologies possessed by NEC but also to implement seamless linkages via the incorporation of a wide range of external services. Linkages with the mission-critical systems of each customer will also become indispensable. The NEC taskforce is determined to continue our current activities in realising our mission.

Reference

Authors' Profiles

YAMADA Yasuhiro
Assistant Manager
Financial Systems Development Division