This paper introduces a generative AI solution designed to enhance operational efficiency and effectiveness for contact centers for enterprises and organizations. This solution functions as a partner for operators. It automates critical tasks for operators receiving calls, such as urgency assessment, case classification, and report generation. It also provides call takers with recommended actions. Furthermore, in the contact center domain, it offers operators real-time call content classification, compliance violation detection, and guided troubleshooting support. This solution, once realized, will reduce manual workload, minimize human error, and enable rapid, high-precision responses to various cases and customer interactions, driving business growth and improving operational outcomes.

1. Background

Amidst rapid changes in modern society, organizations across various fields face heightened demands for more efficient and effective operations than ever before. Across many critical sectors, reports—whether operational or urgent—are becoming increasingly complex and diverse, necessitating fundamental transformation. Similarly, corporate and organizational contact centers confront the significant challenge of addressing diverse customer issues while maintaining strict compliance adherence. The sheer volume and complexity of these demands place a heavy burden on operators, making it difficult to sustain adequate service levels. Under these conditions, traditional methods heavily reliant on manual labor and human judgment often lead to delays in response times and increased human error. Generative AI agents address these challenges not as replacements for humans, but as partners supporting solutions. Generative AI agents collaborate with humans to automate routine tasks, provide real-time insights, and make sophisticated recommendations. This enables operators to work more efficiently, make appropriate decisions, and focus on more complex, high-value-added tasks. This paper details generative AI solutions that enhance human capabilities, address unique operational challenges in high-demand operational centers and corporate/organizational contact centers, and drive business growth. ^1)-4)

1.1 Command Centers

Mission-critical operations are evolving daily, with an increasing number of incidents spanning multiple domains—public health, mental health, crime, and more—regardless of whether they are operational or urgent calls. This growing complexity places significant strain on operators, who must rapidly and accurately assess the situation, document incidents, and allocate resources under immense pressure. Furthermore, reliance on traditional handwritten notes and rule-of-thumb judgments can lead to significant response delays due to human error and slow decision-making. This AI-driven solution addresses these challenges by providing operators with real-time support for situation assessment, case classification, and record creation, while also automatically capturing essential information during calls. Simultaneously, it assists operators by suggesting recommended actions, enabling the swift and precise mobilization of optimal resources.

1.2 Contact Centers

Corporate and organizational contact centers serve as the frontline for customer interactions, handling diverse technical issues that demand swift and accurate resolution. The efficiency of this process directly impacts customer satisfaction and loyalty. Key challenges include consistently classifying diverse inquiries, early detection of compliance violations, and the ability to properly guide operators through complex problem-solving workflows. This solution is specifically designed to enhance the capabilities of both operators and supervisors. For operators, it automatically classifies call content, detects customer dissatisfaction, and suggests optimal follow-up questions. This effectively narrows down the root cause of issues and accelerates the resolution process. For supervisors, it detects compliance violations in real time—such as employees mistakenly sharing confidential information—enabling immediate intervention and corrective action. This dual approach ensures more efficient and consistent call handling while guaranteeing adherence to internal policies.

2. System Overview and Architecture

This system is a comprehensive human-in-the-loop generative AI platform designed to enhance operator capabilities in public safety and corporate/organizational settings. As shown in Fig. 1, this architecture processes voice input from citizens in real time, integrates information, and provides operators with actionable recommendations and summaries. This is achieved through the coordinated operation of multiple AI agents performing specialized processing while interacting with knowledge bases both inside and outside the organization.

This architecture ingests voice input from citizens as raw data into the system. The audio stream is passed to a Speech Recognition AI Agent, transcribing the conversation into text format. This transcribed data becomes the primary input for subsequent processing.

The text-based input is passed to the core processing pipeline. First, the Information Extraction AI Agent identifies and extracts key entities, facts, and relationships from the transcribed text. For example, in a public safety context, information such as location of occurrence, type of incident, number of people involved, and special requirements is extracted. The extracted information is stored in working memory, serving as dynamic contextual information for the ongoing call.

Next, the Context Search AI Agent accesses the working memory and interacts with two critical data sources. One is the Classification and Situation Assessment Description Database, which contains predefined categories for incident classification and systematic frameworks for situation assessment (e.g., situation assessment models). The other is the External Knowledge Database, which provides information such as geographic data, historical incident logs, and technical specifications to supplement the context. The Search AI Agent acquires relevant knowledge based on the extracted information, temporarily storing it in working memory to build a richer context.

The context integrated into the working memory is utilized by the Situation Assessment AI Agent and the Incident Summary AI Agent. This constitutes the core of the system's decision-making. Based on the integrated information, AI agents generate real-time situation assessments (e.g., situation labels) and create primary incident summaries. This process includes the Grounding and Verification step shown as dashed lines in the diagram. It ensures accuracy and reduces the situation of hallucinations by cross-referencing system outputs against predefined rules and facts.

The final output includes situation assessment recommendations, explanations, and a summary, which are presented to the operator. This human-in-the-loop design is critically important. Operators review the system's output, provide feedback as needed, and make the final decision. This feedback loop enables the system to continuously learn and improve. Ultimately, the verified information is sent to a log archive for historical management. This completes the processing cycle, ensuring all data is properly recorded for future analysis and learning.

The core of the system proposed in this paper is a multi-AI agent network that mimics human collaboration, with each AI agent dedicated to specific cognitive tasks. As shown in Fig. 2, this network consists of three main AI agents—the Information Extraction AI Agent, the Inference and Planning AI Agent, and the Smart Memory AI Agent that collaborate with each other.⁴⁾⁵⁾ This architecture enables each component to function integrally based on the outcomes of others, providing operators with comprehensive and intelligent solutions.

Information Extraction AI Agent: This serves as the system's "eyes" and "ears," capturing raw unstructured data and converting it into structured formats. Its primary roles include:

Inference and Planning AI Agent: This serves as the network's "brain," analyzing information and making decisions. It receives structured data from the Information Extraction AI Agent and performs complex cognitive tasks. The primary roles of this AI agent are twofold:

Smart Memory AI Agent: This serves as the network's "central nervous system," functioning as a dynamically shared knowledge base. Its role is to enable inference and planning AI agents to access both short-term context and long-term knowledge. Its components are as follows:

This integrated network of specialized AI agents ensures all data is processed by the appropriate component at the right time, enabling a powerful and seamless workflow. This enhances the responsiveness of human operators.

3. Core Technology Description

The success of this solution is underpinned by a modular, highly integrated core technology stack. Section 3 details the key technical components that work together to enable the system's advanced capabilities, from real-time data processing to intelligent decision-making. It further delves into the architecture of the AI agents and their collaborative role in driving contextual understanding and proactive actions.

3.1 Core Technology 1: Speech Recognition

NEC has been engaged in research, development, and commercialization of speech recognition technology since its early days, consistently tackling the challenging task of interactive speech recognition. We have developed automatic transcription technology for government agency meetings, deploying commercial solutions that streamline meeting minute creation.⁸⁾ This technology is also utilized to support lay judge trials⁹⁾ and assist contact center operations.¹⁰⁾

In recent years, rapid advances in deep learning have revolutionized speech recognition technology. Recognition accuracy has improved dramatically and is now considered comparable to human listening ability. NEC has also evolved its speech recognition technology by leveraging the latest deep learning methods, commercially deploying it as "NEC Enhanced Speech Analysis"¹¹⁾ (hereinafter referred to as NEC Speech Recognition) for business support services.

NEC has developed proprietary technologies to achieve practical speech recognition for real-world environments. 

This improves recognition accuracy for business conversations and also enhances robustness to workplace noise and speaker identification (Fig. 3). The following details the key features of these technologies.

(1) Accuracy Improvement through Deep Learning

NEC Speech Recognition utilizes the latest deep learning technology to perform high-precision processing from feature extraction of audio signals to conversion into text strings. Previously, deep learning was applied separately to speech processing and language processing before integration. This has been replaced with a single neural network optimized for the sequence conversion from speech to text. By employing an end-to-end training method to optimize the entire process, accuracy has been enhanced. Furthermore, we have streamlined speech recognition processing by developing algorithms that support sequential processing and reducing the computational load of neural networks,¹²⁾ enabling "real-time" text output with minimal latency.

(2) Adaptation to Business Conversations

NEC Speech Recognition achieves high accuracy in conversations specific to business environments. To handle the conversational styles and business-specific expressions frequently used in professional settings like negotiations and meetings, NEC Speech Recognition constructed a large-scale conference recording dataset. By training the speech recognition system using this data, recognition accuracy was significantly improved, achieving a level suitable for practical business applications.

Furthermore, business conversations frequently involve terminology specific to certain industries or specialized fields. To address these domain-specific terms, NEC Speech Recognition developed a customization feature that allows relevant materials, such as operational manuals and presentation materials, to be directly incorporated into the speech recognition system. The effectiveness of this approach has been confirmed through proof-of-concept experiments targeting subtitle generation scenarios for live broadcasts.¹³⁾

(3) Adaptation to Business Environments

NEC Speech Recognition incorporates features enhancing adaptability to diverse operational environments. Specifically, to reduce ambient noise affecting recognition accuracy, we developed noise filters that effectively suppress various noise types. By leveraging deep learning, we improved speech extraction accuracy and integrated it as a preprocessing step in the recognition pipeline. Furthermore, to address quality degradation caused by audio compression in remote meetings, we trained the system using the aforementioned meeting recording dataset.

It can also accurately identify "who said what" in conversations involving multiple speakers. This capability utilizes speaker recognition, a biometric authentication technology based on individual voice characteristics. NEC has developed technology capable of identifying speakers even within natural conversations, achieving 95% authentication accuracy in third-party evaluations conducted by the U.S. National Institute of Standards and Technology (NIST).¹⁴⁾

3.2 Core Technology 2: Context Understanding

The operational effectiveness of this system hinges on its ability to understand and utilize conversational context in real time. This capability is realized through two core technological components: the Information Extraction AI Agent and the Smart Memory AI Agent. By collaborating to transform unstructured conversational data into structured, actionable insights, these AI agents overcome the inherent limitations of conventional generative AI models. 

Traditional information extraction methods merely extracted explicit keywords directly. NEC's approach, however, employs the Information Extraction AI Agent. It is designed to extract and understand not only explicitly stated content during speech but also implicit information and the underlying intent behind requests. Relying solely on explicit extraction led to the challenge of responses like "No information available" in cases requiring contextual understanding or access to external data.

This solution addresses this challenge by implementing a multi-step, AI agent-driven workflow, as shown in Fig. 4. For example, when an operator asks, "What is the status of Case #200-001?", the system does not merely search call records for a direct answer. Instead, it executes the following advanced process:

This AI agent-driven workflow enables the system to provide comprehensive responses like "Case #200-001 is ongoing. Unit 3 has arrived on-site. The latest update at CEST 2:15 reports smoke visible from the third floor," rather than simply replying "No information available." Key elements enabling this are:

Thus, the Information Extraction AI Agent functions as a specialized large language model (LLM) with advanced function call capabilities. When processing transcribed text, it does not merely rely on internal knowledge. Instead, it first identifies key entities such as names, dates, and case IDs. It then executes predefined functions using these as parameters, interacting with external databases and systems. For example, after understanding an operator's request, it can automatically issue an API call to a specific database function (e.g., ‘get_incident_status(id)’). This autonomous orchestration of function calls and external data retrieval by the agent enables it to provide comprehensive, data-driven answers that go beyond simple text-based responses. This multi-layered approach ensures operators receive highly accurate and relevant information.

Ultimately, the Information Extraction AI Agent functions like a professional assistant, "listening" to calls in real-time to record key points and present extracted information to the operator. Furthermore, it automatically links critical data such as names and phone numbers to existing databases, streamlining record-keeping and information retrieval.

The Smart Memory AI Agent builds upon the foundational capabilities of the Information Extraction AI Agent, providing the essential context and long-term knowledge necessary for the system to perform effective reasoning. While the Information Extraction AI Agent excels at extracting and organizing unstructured data from calls in real time, the Smart Memory AI Agent grounds that information through a persistent, domain-specific knowledge base. This two-tiered approach enables the system not only to understand what is said in the moment but also to grasp it within a broader, more accurate context. Consequently, it can provide operators with reliable, evidence-based recommendations.

In particular, standard LLMs have a fundamental limitation: short-term and limited memory. This can make it difficult to maintain context or consistently follow instructions during lengthy conversations. Smart Memory AI Agents are designed to solve this challenge, enabling AI agents to effectively memorize, manage, and utilize operational knowledge.

As shown in Fig. 5, the Smart Memory AI Agent integrates a Knowledge Manager with short-term and long-term memory components, delivering key benefits:

To realize these benefits, the knowledge manager of the Smart Memory AI Agent incorporates the following capabilities:

Thus, the Smart Memory AI Agent functions as an orchestrator, centrally managing the flow of information between the LLM and the knowledge base. This agent utilizes an advanced memory management system to constantly monitor conversation content, determining when to store new information and when to reference existing knowledge. This system clearly distinguishes roles: short-term contextual information held during a call is stored in working memory, while long-term operational knowledge resides in long-term memory. The Smart Memory AI Agent indexes and retrieves the most relevant data from the long-term memory. This ensures LLM responses are not only contextually appropriate but also fact-based and aligned with predefined operational rules. This integration and management of different memory components overcomes the inherent memory limitations of LLMs, enabling long-term dialogues and large-scale knowledge-based reasoning.

The core technology of this solution is formed by the collaborative operation of the Information Extraction AI Agent and the Smart Memory AI Agent. This enables three key functions: Situation Assessment, Smart Record Creation, and Call Classification. By working together, these two AI agents transform raw conversation data into actionable insights, significantly enhancing the operator's judgment and response capabilities.

In conclusion, the complementary collaboration between the Information Extraction AI Agent and the Smart Memory AI Agent builds a robust framework supporting contextual understanding. The Information Extraction AI Agent functions as a perception engine, converting unstructured data like voice conversations into structured data in real time. Meanwhile, the Smart Memory AI Agent serves as a cognitive foundation, providing long-term knowledge and contextual awareness. This dual architecture of two AI agents enables system outputs—such as situation assessment, case summarization, and call classification—to be generated not merely based on keyword extraction, but as the result of advanced analysis grounded in operational knowledge and responsive to the subtle nuances of human interaction.

3.3 Core Technology 3: Action Decision-Making

The final element of the core technology is the Inference and Planning AI Agent, which transforms the contextual understanding provided by other AI agents into actionable steps. This AI agent addresses two major challenges: the inflexibility of traditional rule-based systems in dynamic environments and the lack of explainability in decision-making processes. By leveraging advanced reasoning capabilities, it ensures that recommended actions are not only timely and appropriate but also that the reasoning behind the decision is explainable and justifiable.

The Inference and Planning AI Agent achieves these objectives through a three-tiered intelligent decision-making approach, as illustrated in Fig. 6.

The entire logical process, from recognizing initial symptoms to determining recommended actions, is fully explainable. The system can trace back the data and analytical processes underpinning its recommendations, providing clear audit trails that answer questions like "Why was a specific resource deployed?" and "Which factors were considered in the decision-making process?". This transparency builds trust with field operators, enabling swift verification of recommendations and easy adjustments as needed.

Thus, the Inference and Planning AI Agent operates as an advanced decision-making engine through a multi-stage, iterative process. It is designed as an orchestrator that leverages multiple specialized LLMs and tools, rather than relying on a single large-scale model. The planning processing module breaks down complex tasks, such as an entire incident response, into a series of smaller, manageable subtasks. For each core subtask, it uses advanced inference processing modules to generate feasible action candidates. It then verifies and executes these actions while calling upon external tools (tools capable of interfacing with databases and various systems). This process repeats in a continuous loop. The AI agent maintains awareness of the current state from its working memory, plans actions, executes them, and updates the working memory with new information as it becomes available. This cycle is controlled by internal logic while being continuously refined by the latest contextual information from the Smart Memory AI Agent. Consequently, it adapts flexibly to real-time fluctuations, such as new information from the caller or changes in traffic conditions, consistently maintaining optimal and explainable recommended decisions.

Through the collaborative framework of the Information Extraction AI Agent, Smart Memory AI Agent, and Inference & Planning AI Agent, the system achieves two primary operational functions: Next Question Prediction and Action Recommendation. These functions demonstrate that the system has evolved beyond mere information processing to exhibit proactive, decision-supporting behavior.

4. Conclusion

The generative AI solution introduced in this paper represents a groundbreaking initiative that supports operators, dramatically enhancing their capabilities. By establishing a collaborative partnership where specialized AI agents and human knowledge/experience mutually complement each other, this system resolves challenges inherent in conventional systems, such as inflexibility and excessive cognitive load on operators. The architecture, based on NEC's speech recognition technology, integrates an Information Extraction AI Agent, a Smart Memory AI Agent, and an Inference and Planning AI Agent. This collaborative framework supports real-time contextual understanding and intelligent decision-making.

This technology delivers clear benefits in real-world operations. In critical sectors, it enables rapid and accurate situation assessment, efficient smart record creation, and automated call classification, supporting more precise and swift responses for both operational and urgent situations. In corporate and organizational contact centers, it supports rapid problem resolution through next-question prediction and optimal response decisions via action recommendations. Furthermore, by integrating advanced reasoning technologies such as complex reasoning, common-sense reasoning, and multimodal reasoning, it enhances the accuracy and reliability of recommendations while ensuring explainability—the ability to clearly articulate the basis for decisions—thereby maintaining human oversight.

This generative AI solution is not merely a tool, but a transformative partner that extends human capabilities, reduces errors, and boosts operational efficiency. By automating routine tasks and providing data-driven insights, it enables human professionals to focus on complex, nuanced work requiring empathy, critical thinking, and rapid decision-making. Consequently, it empowers the delivery of higher-quality outcomes.

References

Authors’ Profiles