February 7, 2023

Solving social challenges with human-behavior summarization from videos

Visual Intelligence Research Lab Media Analysis Platform Research Group
Karen Stephen

Karen Stephen received Bachelor of Technology in Department of Electronics and Communication Engineering from National Institute of Technology Calicut, India in 2016, and Master of Technology in Communication and signal processing from Indian Institute of Technology Bombay, India in 2018. Her Master thesis was on human activity recognition from videos. She Joined NEC on October, 2018.

Focus on understanding and summarizing human behavior from videos

After joining NEC, I have worked on multiple topics, such as detecting human-to-human interaction in videos and recognizing human actions from 360-degree videos. The benefit of 360-degree cameras is its wide field-of-view because of which a single camera can monitor a large area. My task in this project was to recognize actions of humans from 360° camera footage. That was an exciting project.
Since 2022, I have started working on a new research theme called long-term human-behavior summarization from videos, which is currently ongoing.
Nowadays, vast amounts of video data are generated daily-including videos created and uploaded to social media platforms and CCTV(Closed-circuit television) camera footage from many public places like shops, factories, streets, etc. If a user wants to find a particular event of interest from these videos, going through hours of video data is very tedious and time-consuming.
Our technology aims to create a summary (in the form of easily understandable text and a short video) of the events which the user is interested in, from long videos which contains many events . For example, many shop owners would like to know customers' purchasing behavior, but going through long videos takes a lot of work. Our technology aims to create a concise summary video showing all the purchasing events in the video, along with some insights.
This research theme is still new, and we are in the process of realizing this technology.
Our team collaborates with professors and researchers at the National University of Singapore on this project.
We published a paper on this research topic at ACM Multimedia 2022 International conference, and it won the award for the best brave new idea paper.
Link to paper: https://dl.acm.org/doi/abs/10.1145/3503161.3549202

Collaborate with experts to create the future society and myself

My master's thesis that I was writing then was related to video analytics. When I started working in video analytics and computer vision, I heard from my seniors about NEC, a very prestigious Japanese company. And many of my seniors were working in NEC during that time.
NEC came to my university for placements, and they had a job profile that matched exactly withmy Masters thesis work . It was a very good opportunity for meto enter this prestigious company in the same field I was working in.
As a foreigner in Japan, one big struggle is the language. It was one issue that I also faced. NEC provides language training for foreign researchers. Before coming to Japan, NEC had provided me with Japanese language training for about two months. It covered basic Japanese such as how to greet others, how to ask for help , and so on. Even after coming to Japan, I continued taking Japanese classes and it helped to significantly improve my Japanese speaking and listening skills.
NEC wants to expand its business more globally and is very interested in bringing in cultural and gender diversity by having foreign talents. NEC also puts effort into creating an atmosphere that makes it easy for foreigners to work in the company.
NEC's support goes beyond just language training. . NEC is very proactive and offers many opportunities for researchers to grow. For example, the company supports researchers who are interested in attending International conferences, expositions and interacting/collaborating with other researchers.. We also have the freedom and flexibility to choose where we work from, whether at home or in the office.
NEC is a prestigious company and has a long history of more than 120 years. Like today's face recognition, NEC has created several technologies that have received worldwide acclaim. We have many top class researchers who have achieved such results. Therefore, we are confident that we will be able to create groundbreaking technology.
Everyone in my team is very helpful and happy to give feedback or help whenever I need. My research sometimes requires technologies such as human tracking and pose estimation, so I am able to freely collaborate with other members who are experts in that field . Collaborative work with other researchers is possible and is highly encouraged in our lab. Collaborations could be done within NEC or also outside NEC, with Professors working in various universities (academia). This is one of the merits of doing research at NEC.
I find mywork at NEC very challenging, but my research also has the potential to change the future of our society. Through my research collaborations, I continue to learn more and keep improving myself.

Bridging the gap between technology and society

I am currently working as a researcher, but in the future, I want to understand the business side as well. NEC is a company that simultaneously develops technology and business to provide them to society. We want to create technologies that are valuable to society by focusing on its business aspects. There is a gap between technology and society.
Being at NEC will give me the opportunity to learn how to fill that gap. My team's superiors and seniors are involved in business and collaborate with business unit members for tech trials and PoC (proof of Concept). I have yet to have a particular desire to contribute to a specific field, but I would like to see my work benefit the society in future.
I want to tell people like me who aim to become a researcher to work with passion. It takes passion to go a long way to achieve research results. NEC is a company that appreciates that passion. Also, collaboration is powerful. So feel free to discuss your ideas with other researchers or even your family members. You'll get more ideas that way. Be passionate about your research topic, be open to others and empower each other.

A day at work

Message to my past self in my school days