NEC Develops Distributed Processing Technology to Increase the Speed of Big Data Analysis - Hadoop processing for machine learning increased by more than ten times -

*** For immediate use October 8, 2013


Tokyo, October 8, 2013 - NEC Corporation (NEC; TSE: 6701) has developed distributed processing technology that increases the speed of big data analysis. Machine learning (*1) based on big data for purposes such as recommendations (*2), price predictions and demand predictions, conventionally requires a great deal of time. This new technology enables Hadoop (*3) processing to be carried out more than ten times faster, which enables analysis results to be capitalized on more quickly.

At present, big data analysis is usually carried out using a distributed processing framework, the main example of which is Hadoop. However, it has always been difficult to carry out multiple processing operations at high speeds using Hadoop. This includes machine learning aimed at extracting regularity and patterns from data.

If data analysis takes a long time, companies are unable to use analysis results as quickly as they would like. When carrying out advanced analysis of large volumes of data for instance, including recommendations, price predictions and demand predictions, it is necessary to examine processing results in a variety of different ways, to change the axis of analysis and to reprocess data numerous times. Depending on the volume of data, it can take over a week to produce the final analysis results in some cases. This makes it impossible to carry out analysis in real time, and can lead to missed opportunities.

The new distributed processing technology developed by NEC increases the speed of iterative and matrix operations used frequently in machine learning, without any loss of reliability.

Having tested the relevant technology, it has been proven to increase speeds by over ten times when compared to conventional Hadoop processing. This enables analysis results to be used more quickly by shortening analysis times from over one week, for example, to just one day (*4).

In recent years, there is growing demand for more advanced analysis of big data, rather than aggregation and other simple analytical processes. Machine learning is a prime example of advanced analysis, for purposes such as recommendations, price predictions and demand predictions. This new distributed processing technology caters precisely to this demand.

Part of this work was performed with the METI R&D of Industrial Science and Technology Frontier Program's "Research and Development Project for Green Network/System Technology (Green IT Project)," supported by the New Energy and Industrial Technology Development Organization (NEDO).

NEC presents details of this achievement at the IEEE International Conference on Big Data 2013, an international conference in Santa Clara, California from October 6 to 9.

Going forward, NEC will continue to develop technologies that help users to capitalize on the potential of big data.

Notes

(*1) Machine Learning: Technology designed to extract regularity, patterns and intelligence from data, establish a picture of the current situation, and perform predictions for the future

(*2) Recommendations: Method of recommending products or services that are likely to be of interest to the user based on their purchase history

(*3) Hadoop: Open source distributed processing framework developed and published by the Apache Software Foundation

(*4) Based on an example of processing recommendations based on 4 million users, 500,000 products and 20 million purchase history records

About NEC Corporation
NEC Corporation is a leader in the integration of IT and network technologies that benefit businesses and people around the world. By providing a combination of products and solutions that cross utilize the company's experience and global resources, NEC's advanced technologies meet the complex and ever-changing needs of its customers. NEC brings more than 100 years of expertise in technological innovation to empower people, businesses and society.  For more information, visit NEC at http://www.nec.com.


NEC is a registered trademark of NEC Corporation. All Rights Reserved. Other product or service marks mentioned herein are the trademarks of their respective owners. (C)2013 NEC Corporation.