In recent years, technologies related to physical AI and embodied AI have received increasing attention as advancements are made in autonomous driving and bipedal robot autonomy. In addition to AI-driven automation of individual devices, it is becoming ever more important to develop technologies that enable coordination among multiple devices and people, allowing for system-wide optimization. At NEC, we are actively researching and developing real-world agents to achieve such coordination. For real-world agents, it is essential to create technologies that can realistically apply the output of generative AI to actual operations. This paper discusses NEC's work on real-world agents and introduces application examples in the manufacturing and logistics domains.

1. Introduction

In recent years, the level of automation and the range of applications for artificial intelligence (AI) have been expanding rapidly, resulting in a greater impact on society. While traditional AI has mainly focused on numerical and categorical prediction as well as image recognition, the emergence of generative AI and large language models (LLMs) has enabled AI to generate content. Moreover, the scope of generated content is expanding beyond text to include images and even robot actions. More recently, AI agents that can perform tasks on behalf of humans have also been attracting attention.

In the context of applying AI to the control and optimization of actual systems, technologies such as physical AI and embodied AI—aimed at achieving autonomous driving and bipedal robot autonomy—have attracted attention.¹⁾ At NEC, in addition to automation technologies for individual devices using AI, we believe that, going forward, it will be increasingly important to develop technologies that enable coordination among multiple devices and people, optimizing the entire system as a whole. This paper introduces real-world agents utilizing LLMs, a technology NEC is actively researching and developing to realize such coordination.

Real-world agents are expected to optimally automate or semi-automate operations in physical settings such as factories and warehouses, while appropriately assisting human operators. However, because real-world agents apply AI to actual operations, it is not possible to simply use outputs from LLMs that may appear plausible but are actually incorrect—a phenomenon known as hallucinations—in real-world contexts. This paper describes mechanisms for obtaining responses from AI that can be reliably applied to actual operations, as well as application domains where the deployment of real-world agents is anticipated.

2. Challenges in Applying Generative AI to the Real World

To automate real-world operations, it is necessary to formulate plans that satisfy high levels of reliability and safety within a practical timeframe. Even when incorporating generative AI models such as LLMs, solutions to complex optimization problems—entwined with physical constraints and dynamic characteristics—must be generated in real time, with logical correctness and feasibility for actual execution in real-world settings.

When using generative AI to create plans for executing tasks in the real world, the feasibility of the proposed solutions becomes a major concern. Generative AI models such as LLMs possess extensive and diverse knowledge and may be able to generate plans for tasks that are difficult for traditional planners to handle. In fact, a number of planning techniques utilizing LLMs have already been proposed. However, while these models may output plans that appear correct at first glance, they may not fully consider task conditions, environments, or the physical constraints of robots, resulting in a lack of guaranteed reliability and safety in real-world operations. For instance, in robotic automation, physical constraints can prevent robots from executing planned actions, or may lead to issues such as collisions with obstacles. These challenges become increasingly pronounced as the complexity of the tasks and the scale of the environment grow.

Furthermore, the quality of solutions generated by generative AI is greatly influenced by the structure and content of the text prompts used to instruct the AI. Therefore, designing optimal text prompts for complex real-world problems is itself a significant challenge. Real-world operations often involve numerous constraints—based on specific task conditions and physical laws—that are difficult to fully express through text. To create plans with sufficient reliability and safety, it is necessary to iteratively verify the generated solutions and refine the text prompts as needed through repeated interaction with the AI. However, conducting this trial-and-error process manually is impractical, making it difficult to generate executable real-world plans within a practical timeframe.

3. Real-World Planning Automation Framework Using Generative AI: Real-World Agents

NEC is researching and developing real-world agents, a framework designed to generate practical plans for actual applications by enabling multiple LLMs to collaborate in reasoning (Fig. 1). Real-world agents continuously verify the feasibility of plans and iteratively optimize the overall plan based on these results.

This framework is composed of an orchestrator LLM, which oversees the entire process, and individual LLMs responsible for handling subtasks assigned to entities such as people or robots that act in the real world. The orchestrator LLM grasps the overall objectives and the structure of the tasks, divides the main task into subtasks, and assigns them to the respective individual LLMs. Each individual LLM then generates a plan for executing its assigned subtask. The generated plans are then validated for feasibility using action prediction models, including physics simulators, world models that incorporate real-world knowledge and physical constraints, and other relevant tools. The results of this validation are fed back to the orchestrator LLM, which automatically refines the prompts and generates more appropriate task divisions and assignments based on this feedback. By repeating this sequence of processes, the framework ultimately produces an optimal plan for the entire system.

With this architecture and process, it is possible to automatically generate feasible solutions even for large-scale real-world tasks. By configuring the roles of the LLMs according to the specific domain, the framework can be applied to automate real-world actions and operations across a wide range of fields.

4. Applications of Real-World Agents

4.1 Manufacturing domain

In large-scale industries such as automobile manufacturing, a single product is completed through several hundred multi-stage processes. Under such complex process control, designing workflows that optimally allocate tasks—such as welding and painting—while meeting cycle time requirements for each stage has traditionally required time-consuming and labor-intensive trial and error by human operators. In the welding process in particular, thousands of welding points must be distributed across dozens of stages. If strict constraints such as robot reachability or cycle times are not met, the allocation must be redone. As a result, conventional process planning has required a significant amount of work and resources.

Real-world agents can be applied to address these process design challenges (Fig. 2). The orchestrator LLM oversees the allocation of welding points and the adjustment of each process stage. Individual LLMs are responsible for specific process stages and coordinate information exchange with the orchestrator LLM. Each individual LLM evaluates factors such as robot placement, the feasibility of welding points, and cycle times for its assigned process stage. If requirements are not met, the results are fed back to the orchestrator LLM, which then reassesses and optimizes the allocation. This iterative process continues until all specifications are satisfied. Furthermore, real-world models are used to predict and evaluate robot placement feasibility, welding point success, and cycle times, while cost functions support evaluation and optimization. Principle-level evaluations within NEC have indicated that, by introducing real-world agents, tasks such as process planning and operation verification—which previously required over a month of manual work—could potentially be completed in just a few days. In this way, real-world agents demonstrate significant utility in automating and streamlining manufacturing process design.²⁾

4.2 Logistics domain

In recent years, labor shortages in logistics warehouses have become a serious issue, making the reduction of manual labor and the improvement of operational efficiency key challenges. In response, various automation solutions—such as the use of automated guided vehicles (AGVs) for automating picking operations—are increasingly being implemented. However, when robots are introduced into conventional warehouse environments, the sites are often irregular and dynamic, making it difficult for multiple robots to move safely without interfering with each other, and to adaptively handle goods that are arranged in a non-uniform manner.

By applying real-world agents to these challenges, it is possible to achieve automatic scheduling for multiple robots. The orchestrator LLM decomposes tasks for each robot based on shipping orders. Individual LLMs then use world models to evaluate aspects such as collision avoidance between robots, the robots’ ability to grasp items, and actual working times (takt time), enabling adaptation to irregular and dynamic environments. If any issues are detected, this information is fed back to the orchestrator LLM, which then reassesses and optimizes the path planning.

In this way, by repeatedly cycling through hierarchical planning and feedback, it is possible to dynamically generate feasible path plans that are adapted to the actual site environment. To verify the effectiveness of the proposed method, comparative experiments were conducted on picking tasks in automated warehouses, comparing it with conventional LLM-based path planning approaches. As a result, it was confirmed that the real-world agent improved the task success rate from 40% with conventional methods to 95%. (Fig. 3).

4.3 IT services

In the operation and maintenance of IT services, support personnel respond to a large number of incidents reported by users. Traditionally, operations and maintenance managers have had to allocate tasks while considering each staff member's skills and workload, as well as engaging in repeated communication with users to identify the root causes of issues. As a result, there have been challenges in optimizing task assignments and reducing lead times for problem resolution.

Real-world agents address these challenges by automating the operation of IT maintenance services. The orchestrator LLM searches a historical response database for similar incidents whenever a new incident occurs, determines the appropriate response measures to take, and estimates the associated response costs. Based on these results and the current task allocation status of each staff member, it automatically performs optimal task scheduling. Additionally, if insufficient information is available for incident resolution, the system is equipped with a process to automatically obtain additional details from users, thereby streamlining issue identification. The individual LLMs are assigned to each staff member, predicting each person's skill level based on past response history and assessing the feasibility of executing the assigned tasks. If a task is determined to be difficult to handle, the individual LLM provides feedback to the orchestrator LLM, which then re-schedules the tasks. By repeating this process, the optimization of task allocation can be achieved.

With the automation enabled by real-world agents, it is expected that both the workload of staff and managers in IT service maintenance and operations can be reduced, and that the lead time for incident resolution can be shortened.

5. Conclusion

Real-world agents have the potential to fundamentally transform AI applications in real-world domains, which until now have focused primarily on tasks such as equipment failure detection and inspection of shipped products. Real-world agents suggest that AI-driven automation can be applied even to highly flexible operations—such as optimizing factory production scheduling or coordinating tasks across multiple devices in logistics warehouses—which have traditionally relied on human trial and error and collaborative adjustment.

Going forward, it will be essential to conduct verification and further technical refinements in real-world environments such as factories and logistics warehouses to advance the practical implementation of real-world agents. Through the development of such technologies, the advanced integration and collaboration of AI with industrial systems is expected to contribute to the realization of a more prosperous society.

References

Authors’ Profiles