March 21, 2024

Physical motor function is fundamental for healthy longevity and vibrant living. Nevertheless, many people today are troubled by physical problems such as lower back pain. While regular motor function care is essential for muscle movement and posture improvement, physiotherapists’ motor function care at hospitals and clinics are not something readily accessible. Focusing on this issue, self-care support AI technology is described as a technology that significantly improves the quality of motor function self-care. We spoke with our researchers about the purpose and details of this technology.

Providing physiotherapist level motor function self-care at home

—What kind of technology is self-care support AI technology?

Kosaka: This AI technology assists people in maintaining and improving motor function while at home. Based on images of a user performing forward/backward bending and rotational flexing taken with a camera on a smartphone or tablet computer, this technology provides an exercise program customized to each individual after the user answers a check list type questionnaire. The main feature is that it assists with quality motor function care on a regular basis. It is packed with physiotherapeutic know-how of professional rehabilitation service for recovery and maintenance of motor function. The technology focuses on lower back pain, which is the leading subjective symptom of illness or injury currently in Japan. Particularly targeting chronic lower back pain (CLBP), it is now capable of identifying the cause and recommending exercise programs.

In the backdrop of the development of this technology is the rapid aging of population toward 2040. Amid increasing medical and long-term care needs, the Japanese government is presenting a national policy for establishing a service provision framework for maintaining and improving motor function. What is most emphasized is the importance of approaching the issue from both aspects of quality and quantity. For example, while we can currently receive quality motor function care from medical professionals such as physicians and physiotherapists by visiting hospitals and clinics several times a month, it is not realistic to receive this service on a more regular basis. It is difficult to have sufficient “quantity” through hospital visits due to distance, time, and economic constraints. The new technology works around this issue by assisting with motor function self-care, which is considered to be important for increasing “quantity.” Having said that, it is still quite difficult to practice correct self-care for motor function without specialized knowledge. In other words, it will become difficult, in turn, to ensure “quality.” It is extremely difficult to achieve both quality and quantity. We then resorted to applying AI technologies to solve this issue.

In a broad categorization, our technology has three major functions. First, it has the function to evaluate the condition of each body part based on camera images. With these images, the AI evaluates the externally-observed condition of each body part. This includes, for example, shapes of movements such as insufficient, moderate, or excessive joint flexion.

The second function is a reasoning function that explores the possible causes of physical problems based on information from questionnaires in addition to camera images. This enables deeper investigation into the cause of the problem by using information from questionnaires alongside visual information. For example, when finding the cause of CLBP, while visual information can only tell things like excessive lumbar flexion, the addition of questionnaire-based information enables reasoning that derives suggestions such as the main cause of CLBP being in the neck.

The third function presents exercise programs that solve the problems. This function can recommend highly-specific exercise programs catered to each person’s individual condition at the time.

These functions were developed through a collaboration of three research teams in NEC. Initially, we, the team that mainly handles healthcare, were working on this alone, but after we found that we also need human pose estimation technology and autonomous reasoning technology to achieve our target, we invited the teams that study these domains for transversal research. I think it’s a unique advantage that NEC has that enables such speedy research. You just hop on the elevator to get to a different floor where there are researchers studying other fields, and call out to them to join the research. We were able to take advantage of NEC’s strengths that have been built up through proprietary research over a wide range of fields.

Creating precision 3D human body keypoint models by simply capturing the entire body with a smartphone camera

――The first function evaluates the condition of each body part based on camera images.

Kosaka: Yes. There are two key technologies that made this function possible. One is the 2D/3D human pose estimation technology, and the other is the posture condition recognition technology.


Ikeda: The 2D/3D human pose estimation technology can create a 3D human body keypoint model in real-time based on human images captured on camera. It extracts joints as three-dimensional key points in space and connects them together with lines to model human movements. There are actually other technologies out there similar to this, but our new development makes a departure from others in the following two points.

First of all, it can casually generate a human pose model from common camera images taken with a smartphone. No special camera is required―a single smartphone camera suffices. We aimed to make it easy to use for anyone at home as our original concept, so we created a system that can be readily used with a smartphone camera. To build this technology, we developed an AI learning method that is robust to different capture angles. This saved the need for strict camera position adjustment. You only need to set up your smartphone so that you are in frame and focus to work this function. Even if the camera shifted during filming, it can keep following you with no problem and continue estimating your pose.

The other point is that we created a model specializing in rehabilitation. A highly-accurate recognition is required in rehabilitation, with a margin of error of one bone or less, so we built an original AI capable of accurate recognition.

With rehabilitation, it is also essential to evaluate body “twist,” but a body twist makes half of the body hide behind the back, which means that precise estimation of depth is required. It was conventionally believed to be almost impossible to create a 3D model from 2D camera images. Therefore, we worked around this problem by creatively teaching our original AI (deep learning).


Ishii: Yes. The first problem that we encountered was how to collect learning data. In deep learning, the volume and quality of learning data directly affects accuracy. So what we did first was set up a filming studio and capture each member of a large cast using multiple cameras surrounding him or her. After filming, we needed to teach the AI where the correct joint points are. To do this, the positions of joint points of the person in the captured image must be annotated in 3D. If done manually, this is an extremely time-consuming task. We developed an automated annotation technology to create data for teaching the AI. Particularly for core areas such as the waist, we prepared meticulous data.

Furthermore, in order to obtain precise depth information, we created a learning method that focuses on the relationship between the various body parts and the camera. This made possible the precise estimation of body keypoint structure that could robustly respond to body twists. Consequently, this also led to the development of human body keypoint estimation that does not rely on certain shooting angles.

Posture condition recognition incorporating expert knowledge

――Based on that 2D/3D human body keypoint information, the posture condition recognition technology is applied to evaluate the condition of body parts.

Noyori: In addition to the 2D/3D human body keipoint information, our original technology that observes the curve of the spine from camera images that captured forward bending, backward bending, and rotation plays a part in the evaluation of each body part. The cause of CLBP cannot be explained without looking at any curves (or lack of curves) in different regions of the spine in detail in addition to skeletal structure. Conventionally, this has been evaluated by using X-rays and other medical equipment that is placed along the spine. With our new technology, we can capture the details of curves in the spine from standard camera images. Comprehensive examination of these 2D/3D human body keypoint information and the curves in the spine is making it possible to produce precise evaluation of the condition of each body part.


Suzuki: Since it was crucial to derive the interpretations for the results of evaluation of each body part, deep learning is not used in this technology. Deep learning is a black-box AI technique that makes the basis behind the output decision unclear. In place of deep learning, we utilized the knowledge and know-how of the professors and physiotherapists of the Tokyo Medical and Dental University, our partner in this collaborative research. An example of a piece of knowledge that we got is, for rotational movement, failure of rotation of a certain body part is characterized by compensatory motion to make up for that rotation. Also rotational movement, which is a three-dimensional motion, tends to result in slightly unstable human pose estimation compared to a more two-dimensional motion such as forward and backward bending. To address this problem, we used knowledge related to human skeletal constraints during rotational movements, including which body parts mainly move and which body parts do not move as much, to automatically correct errors that occur during the extraction of body points in 2D/3D human pose estimation technology. This achieved improved accuracy in the evaluation of the condition of each body part.


Noyori: This study is a collaborative effort with the Tokyo Medical and Dental University. This partnership allowed us to adopt the medical knowledge of doctors and the physiotherapists affiliated with NEC Karada Care (Note 1). For example, in case of forward bending, how the movement of a certain body part should be identified and what is the threshold for correct movement. We discussed such matters one by one while establishing evaluation criteria for the information obtained from camera images. Since this is an unprecedented attempt, there is yet to be any model that integrates such information in a scrupulously detailed manner. We built an AI that precisely evaluates the condition of each body part while carefully hearing out the tacit knowledge the medical professionals have.

Accurately estimating the underlying cause of physical problems

――The second function is a reasoning function that explores the possible causes of physical problems based on video images and the results of questionnaires.

Kawada: Yes, that's correct. In short, this function explores the underlying cause of CLBP while referring to the information obtained from questionnaires, such as which part of the body hurts during what occasion or movement, in addition to the posture condition recognition results. This function is achieved by using the abductive reasoning technology. This is NEC’s proprietary technology that presents reasonable hypotheses based on given evidence in line with a large rule base. For the purpose of acquiring acceptable rationale, this function adopts a system that enables tracking of logic decisions, instead of being a black box.

Specifically, we interviewed physiotherapists and heard out what they were thinking in their heads regarding CLBP and organized them into rules. We verbalize the framework of rules required by abductive reasoning and the framework of rules required in physiotherapy, and repeatedly coordinate and ensure consistency between these two frameworks. We established the rules through numerous discussions with physiotherapists regarding questions such as how to define the issue of “insufficient back flexion” using the physiotherapeutically supported “kinematic issue” framework and how to define the location of pain and conditions. The number of rules we created reaches as many as around 1,500. We aimed to recreate the physiotherapists’ process of questioning a subject, making a hypothesis, and drawing conclusions. It may be close to the so-called “expert system.” However, as a major technical feature, this technology skillfully converts the CLBP issue into a mathematical optimization problem to accelerate solution output in order to handle large rule sets using modern technologies.

The process of how the result was presented in the estimation of cause using this technology can be shown visually in the form of a network model using nodes and links. This further leads to the inference that where the links concentrate is the factor that is closely connected to the medical problem. It is also a great advantage that it lets you know at a glance which issues in different body parts are correlated. Lower back pain often concerns not only the waist, but also the shoulders and knees. Clarifying the mutual effects throughout the entire body can significantly improve the accuracy of reasoning and interpretability.

Along with diagrammatic representation, the new technoplogy also provides a summary function through an LLM. Presenting what can be interpreted from the diagram and what kind of state the user is in in natural language achieves improved user-friendliness and usability.

Recommending customized exercise programs with sample videos

――The third function is recommendation of exercise programs optimized to each individual’s condition.

Kosaka: Yes. It does not simply recommend typified exercise programs, but can recommend the customized exercise programs with sample videos based on a comprehensive review of each user’s condition at different times and previous exercise programs. The provision of sample videos helps each CLBP subject work on the exercise programs on their own while checking the movements at home. I thought this technology was ready to be released when this recommendation function was completed, but then I gave second thought and it came to me that while some people can correctly perform the exercise programs, there are others who can’t do them. In order to provide this technology’s services to a wider user base, I rethought that it is also necessary to provide feedback to ensure or reinforce correct exercises. With that being the situation, we started on the development of a new exercise coaching function. Karen is in charge of this development.


Karen: Yes, I am. With the exercise coaching system that I am currently developing, a user can upload their video that they took of themselves exercising and have the system analyze the exercise in the video and provide them with written feedback. A feedback can be something like: “Keep your legs slightly closed” or “Hold your arms/legs at a 90-degree angle.” The system may also give comments like “Great job on keeping your body trunk straight!” by detecting correct motion. By receiving such feedback, users will be able to perform correct exercises even when they are on their own at home.
This feedback is not a fixed message that is displayed according to a set of pre-defined conditions, but the wording of the feedback is generated using an LLM, personalized to the individual user’s condition at that time, thereby improving the user’s motivation.


――How does this technology determine whether a movement is correct or not?

Karen: We teach the AI to detect if the movements are correct or incorrect by training it with labeled exercise video data that includes both correct and incorrect exercises. Having said that, as mentioned previously in the explanation of other technologies, the bottleneck in the development of exercise-coaching AI was how to collect quality data. The exercise-related data sets that are made public are primarily videos of exercises that involve large body movements, such as yoga and workout programs at gyms. We are dealing with exercise programs centering on rehabilitation, which tend to have smaller movements. This means that we can’t just apply these existing data sets. So, we are currently taking guidance from NEC Karada Care physiotherapists to prepare our own data.


Kosaka: NEC Karada Care physiotherapists are teaching us what exercise movements are correct and what movements are wrong. We learn that and return to the lab to demonstrate it to other researchers. Karen and I have gotten pretty good at performing these demonstrations (LOL). Then we have others do the exercises to create data of correct and incorrect moves. While this may be a somewhat low-key effort, it is one of the unique approaches that NEC can take thanks to its business of running a shop with the Tokyo Medical and Dental University. We are currently preparing original quality data while learning detailed know-how from physiotherapists in order to improve the accuracy of this study.

Preparing for demonstration as personalized service

――Please tell us about the future prospects and goals for this technology.

Kosaka: We are thinking about doing demonstrations of a service using this technology starting next fiscal year. It is critical to get it out on the market if we are envisioning commercialization. We can also accumulate data by having people actually use the service. This will further lead to refining the technology and solve our current problem of not having enough learning data at the same time. Of course, we need to ensure strict management of personal information before we put this out to the real world, which is our primary goal.

On another note, we are also considering to expand this service to overseas markets. Especially in the case of Thailand and other Southeast Asian countries, there just aren’t enough physiotherapists. There are people out there that want quality service but aren't getting it. I hope to bring this solution over to help people maintain good health.


Noyori: Mr. Kosaka just mentioned data accumulation due to this technology being adopted in a commercialized service, and I want to add that I want to use that data more efficiently from a technical perspective. As an example, for annotation, we were having a researcher at the Tokyo Medical and Dental University spend a lot of time on labeling. As the volume of data gets larger, this will not be realistic―we need to think of a workaround like being selective about which data to work with. For example, if the joint keypoints and the line of spine output by our engine are already correct, there is no need for correction. At the same time, there is no guarantee that all of accumulated data is useful. In light of such circumstances, we would like to discuss another technology that can streamline annotation from various approaches and prepare the new technology for future practical application.

Suzuki: I want to research a technology that can efficiently develop a posture recognition system which is aimed at medical cases other than CLBP. We have spent a lot a time and cost to develop current technology, listening to physiotherapists and the professors of the Tokyo Medical and Dental University about forward bending, backward bending, and rotation movements. As it is necessary for the accurate evaluation of condition, the respective recognition processes for forward bending, backward bending, and rotation are designed with great detail. However, if we were to build a recognition process for another motion in the future, we would need to create it from scratch. Thus, for example, we would need to look into the development of new technologies or generalized AI that can quickly be applied to the recognition of other movements.


Ikeda: To keep improving the accuracy of the evaluation of condition, we need to increase the volume of information extracted from video images―not just joint points. For example, to observe the delicate movements of shoulders, estimation of specific points on the surface skin of the shoulders will be necessary. To observe the movements of the neck with greater accuracy, we need to estimate each small joint in the neck to examine the overall movement in 3D. Being able to acquire data of surface skin, tiny joints, and human volume in addition to skeletal structure should enable more diverse condition evaluation.

What I would also like to work on is the expansion of the scope of this technology based on robustness technology. The technology currently targets self-care at home. However, it has great potential in terms of being able to recognize three-dimensional skeletal structure in real-time at any shooting angle. I will continue research with other possibilities in mind, such as use in police officers’ outdoor patrol.

Ishii: Like Mr. Ikeda said, I, too, want to try increasing output information. Currently, we can only get skeletal information of a stick person, but developing a technology that acquires body shape information and/or information on skin movements is one of my future goals.

Another goal is to solve the issue of learning data. Considering the current situation that there is a prevailing issue of ensuring the quality and quantity of learning data, a mechanism that enables high-accuracy model teaching even with little or no 3D annotation data may be the key to a breakthrough in future R&D. I hope to develop a technology that is independent of the volume of learning data, incorporating various concepts from small data learning, self-supervised learning, and unsupervised learning.


Kawada: I hope to dig into the expansion of the current knowledge base. For CLBP, we were able to build a knowledge base by verbalizing the knowledge inside physiotherapists’ heads; however, this kind of approach also has the risk of producing tacit knowledge to verbalize the implicit knowledge of physiotherapists.

Therefore, if the scope of application is expanded beyond CLBP in the future, or whenever there is an addition or update of input information such as video images and questionnaire items, how to position that vis-à-vis the existing knowledge base and how to expand it are some issues that we need to keep working on. I believe one approach is to keep the problem simple and concise, avoiding complication whenever possible.

Cooperation of physiotherapists will naturally be essential for this attempt. How can we efficiently draw out their knowledge? How can we expand the knowledge base without causing them too much hassle and be able to attend to various other medical conditions? These are some questions that I would like to think about.


Karen: I want to further improve the efficiency of the development of the exercise coaching system. There are many exercises and even the same exercise can have multiple variations with varying difficulty. For example, for the same exercise, there can be beginner, intermediate and expert variations which might be different from each other. Detecting the correct and incorrect moves for each of these exercise variants will require massive amount of annotated training data which can be a bottleneck in scaling it up to cover a large number of exercises.

So, I would like to explore efficient ways to adapt the AI so that it can detect incorrect moves for advanced variants of an exercise while being trained only on its basic exercise, by utilizing the knowledge about the exercise.


Kosaka: The new technology aims to deliver self-care support service to a wide range of people. I will continue research so that I can contribute to people’s health through providing quality daily self-care.

Members introduction

Go back to Featured Technologies