Accounting for roughly 80 percent of Internet traffic, video data has become a ubiquitous feature of modern life, with the use of video extending to applications ranging from broadcasting to teleworking. What makes all of this possible is video coding technology which compresses large, unwieldy video data, shrinking them down to a size suitable for fast, smooth online distribution. This paper introduces NEC’s video coding technology for ensuring the safety, security, fairness and efficiency of broadcasting infrastructure, and discusses NEC’s initiatives to realizing digital transformation (DX) of the broadcasting industry.

1. Introduction

The origin of NEC’s video coding technology can be traced all the way back to1928 when Japan’s first fax machine — the NE-type phototelegraphic equipment — was developed by Yasujiro Niwa and Masatsugu Kobayashi. At the time, Japan’s major newspapers all used imported phototelegraphic machines, but the new NE-type equipment quickly established its superiority, successfully transmitting photographs of Emperor Hirohito’s Accession Ceremony at high speed and with high picture quality.

Ever since, NEC has been a leading developer of video transmission and coding technologies for telecommunications companies and broadcast networks, contributing to the progress of telecommunications and broadcasting not only in Japan but around the world, introducing powerful technologies such as the world’s first television signal compression device and digital terrestrial broadcast encoder, just to name a few.

In section 2, we will review NEC’s long-standing commitment to supporting the broadcasting sector. In section 3, we will describe some of the changes impacting this sector. Section 4 will summarize the discussion and point to future goals.

2. NEC and the Broadcasting Sector — A Long-Term Commitment

2.1 NEC’s video coding technology

High interconnectivity is essential for video services, so compressed data formats have been standardized by international bodies such as ITU-T and ISO/IEC. Even before standardization was achieved, NEC had been actively involved in the development of video coding technology. Many of the technologies developed by NEC have been adopted as the basis for major international standards¹⁾. To ensure that we can offer products and services that meet high technological required in broadcasting services, NEC offers several key proprietary technologies which are discussed below.

2.1.1 High image quality

Based on the pattern, color, and other elements of the input image, the system optimizes the granularity of the reproduced image (compression ratio) of each region according to how noticeable the degradation is when viewed by the human eye²⁾. Complex patterns, where degradation is less visible, are compressed at a coarse granularity to reduce the data volume, while faces and flat areas, where degradation is more visible, are compressed at a fine granularity to maintain high image quality (Fig. 1).

2.1.2 Low computational complexity

In video coding compliant with international standards, video frames (image planes) are processed by dividing them into blocks. NEC’s technology analyzes the patterns in each image plane region to estimate the optimal block shapes. This facilitates optimal rendering even with limited computational complexity ― for example, a flat area such as the sky can be consolidated in a larger unit, while a detailed area such as the contours of an object can be rendered with finer detail in a smaller unit as shown in Fig. 2. This technology has made it possible for us to develop the world’s first hardware 4K encoder compatible with the High-Efficiency Video Coding (HEVC) international standard and is applicable even to field-programmable gate arrays (FPGAs) available since 2014³⁾.

2.1.3 Low latency

In applications where latency of one frame duration or less is required, conventional technology cannot effectively analyze the visual sensitivity and complexity of the accumulated image data. NEC’s encoding technology, on the other hand, uses an NEC-original stochastic model (Fig. 3) that is able to accurately estimate visual sensitivity and complexity with minimal latency. Because processing of the target area is based on several previous lines of historical data, high-precision estimation of visual sensitivity and complexity can be achieved, ensuring both high image quality and low latency of one frame duration or less⁴⁾.

2.1.4 Highly parallel processing

Video encoding is composed of multiple processing steps whose characteristics differ significantly from one another ― such as motion estimation, transformation, and quantization. NEC’s proprietary highly parallel processing algorithms⁵⁾⁶⁾⁷⁾ can optimally use graphics processing units (GPUs) according to characteristics of the processing being performed. We also have the know-how to maximize the operation rate of each processor even when processing assignments for each region in the image plane are changed, by allocating tasks optimally between heterogeneous processors in central processing units (CPUs) and GPUs (Fig. 4).

2.2 NEC’s products and contribution to society

2.2.1 Distribution encoder

Following advances in 4K/8K camera and display technology, the first 4K test broadcasting system was launched in June 2014 via communications satellite (CS). The system went live in December 2018 with full-scale 4K/8K broadcasting launched via broadcast satellite (BS) and 110-degree CS.

NEC’s 4K/8K distribution encoder complies with the HEVC operation standard and ensures high image quality even at a high compression ratio⁸⁾ by applying the NEC-original technology discussed above. Capable of faithfully reproducing detailed patterns as shown in Fig. 5, NEC’s 4K/8K distribution encoder played a significant role in NEC’s 4K/8K UHD broadcasting equipment’s winning the 30th Radio Achievement Award given by the Minister for Internal Affairs and Communications, which described it as a key component of the new 4K/8K satellite broadcasting system that offers end-users a video experience characterized by a true-to-life “being there” feeling⁹⁾.

2.2.2 Video material transmission codec

A video material transmission codec is used for transmitting video material from a shooting location to a broadcast station in combination with a field pickup unit (FPU) that serves as a wireless station for broadcast operation. Inevitably, due to the basic operation properties of the codec, there will be cases where low latency is required ― such as conversations during a live broadcast or switching video feeds during a live sports broadcast such as a golf tournament, for instance, where both wired and wireless transmission are used (Fig. 6).

NEC’s video material transmission codec brings into balance the high image quality required for video transmission applications and the industry’s lowest-level latency¹⁰⁾¹¹⁾.

Because NEC’s video material transmission codec has rendered low latency and high image quality compatible, it has contributed to the effective utilization of frequency bands when FPU frequency bands were switched from 700 MHz to 1.2/2.3 GHz. As a result, NEC was awarded the 28th Radio Achievement Award¹²⁾ by the Chairman of the Board of ARIB.

3. Changes in the Broadcasting Environment and NEC’s Recent Activities

3.1 Changes in the broadcasting environment

In this section, we will examine NEC’s response to various changes in the broadcasting environment.

3.1.1 Lifestyle changes

Over the past decade or so, smartphone technology has accelerated at an incredible pace and penetration is now worldwide. Accompanied by flat-rate service packages, and, most recently, the impact of the COVID-19 pandemic, this has resulted in a massive shift in how users access video content. For conventional broadcasters, the need to transform themselves into Internet businesses has never been more urgent.

3.1.2 Technology changes

Various broadcasting technologies previously accessible only via dedicated equipment are now becoming widely available thanks to the development of IP-based transmission of broadcasting-quality video¹³⁾, the development and widespread use of general-purpose computing triggered by deep learning, and the development of cloud services.

3.1.3 Policy changes

The progress in international standards and full-fledged dissemination of 5G will accelerate integration of broadcasting and communications and availability of universal services.

3.2 Broadcasters move onto the Internet businesses and initiate DX of their operations

When broadcasters simulcast their content on the Internet, some footage may need to be replaced by preregistered still images if potential copyright issues exist. By making this process highly reliable and fully automatic, we believe we can support both the digital transformation (DX) of that particular application and the broadcaster’s transition to the Internet more broadly.

To achieve this, NEC has developed a system that automatically replaces commercials with preregistered still images at frame accuracy by linking to commercial cue signals from the terrestrial broadcasting master system. We have also developed technology that transmits information to video distribution platforms. The information is needed to replace commercials with still images and the transmission format is compliant with the SCTE-35 standard (Fig. 7).

NEC developed a live simulcast encoder solution that applies the above-mentioned technology and started delivering this system to broadcasters in September 2020¹⁴⁾. Broadcasters need only introduce this solution to their existing broadcasting master system and they will immediately be able to operate Internet-simulcast service. Additionally, NEC’s simulcast encoder can also be used for simulcast transmission of terrestrial TV contents via local 5G in the event of a disaster or emergency¹⁵⁾.

In sum, NEC not only provides broadcasters with the technology and know-how to transition smoothly to the Internet and digitally transform their operations, it also provides measures to ensure safety and security for end users should a disaster occur.

3.3 Expanding and enhancing value

3.3.1 Transition to universal services

The nature of universal services is now under discussion at the Information and Communications Council of the Ministry of Internal Affairs and Communications, where various stakeholders are attempting to hammer out a vision for the future. These discussions have taken on greater urgency as broadcasters shift to the Internet and 5G broadband becomes more widespread, blurring the lines between broadcasting and communications.

NEC will contribute to the transition to universal services through several initiatives, including incorporating broadcast mode transmission technology into 5G facilities to improve the efficiency of capital investment in equipment, providing NEC’s proprietary adaptive video streaming control technology¹⁶⁾ to stabilize services on congested network, and adopting Common Media Application Format (CMAF)¹⁷⁾ (Fig. 8) as a unified format for packaging media segments.

3.3.2 Transition to software and cloud computing

Achieving dramatically higher compression performance than the HEVC standard currently used in smartphones and 4K/8K broadcasting, the Versatile Video Coding (VVC) standard was completed in July 2020. Although VVC is now being reviewed to decide whether to employ it for next-generation terrestrial broadcasting in Japan, the huge amount of processing required for encoding makes its practicality an issue of concern. The amount of processing for VVC encoding is estimated to be about 8 times as much as HEVC. To address this issue, NEC applied the original technology discussed above and promptly succeeded in developing 4K real-time VVC encoder on a single general-purpose server (Photo).

The key takeaway here is that NEC developed cutting-edge technology for real-time encoding of VVC ― the newest international standard ― using software that operates on a general-purpose server. Unlike the dedicated hardware previously offered to broadcasters, this technology offers unparalleled flexibility and scalability, making it easy to add functions and expand services. It will also be available as a cloud service in the very near future, enabling further expansion of services and optimization of equipment costs for broadcasting companies. Furthermore, we think it can also contribute to applications such as large-scale video surveillance and public viewing by linking it with a various engines on a cloud, such as video recognition or data analysis by artificial intelligence (AI) (Fig. 9).

4. Conclusion

As the efforts outlined in this paper illustrate, NEC’s commitment to creating new value in communications for individuals and society remains as strong as ever. We make best use of media and are dedicated to providing people with the tools they need to enjoy an enriched and more fulfilling life.

References

Authors’ Profiles