January 13, 2021

"AI suitcase" for a Symbiotic Society, Bringing Together the Technologies of Five Companies

- Technology to Support Mobility and Communication for the Visually Impaired -

According to the WHO, the number of people with some form of visual impairment is estimated to be as high as 2.2 billion people around the world (*1). With the aim of realizing a society where the visually impaired can live without any discomfort, five companies including OMRON Corporation, Alps Alpine Co. Ltd., Shimizu Corporation, IBM Japan, Ltd., and Mitsubishi Motors Corporation, which are specialized in different fields, have developed an "AI Suitcase" by combining their latest AI and robotics technologies. This article introduces examples of companies working together to improve accessibility (*2) for people with visual impairments and integrating technologies to meet the challenges of social implementation.

*1 Source: World report on vision: https://www.who.int/publications/i/item/world-report-on-vision
*2 Accessibility: ease of access to information and services

Enabling Free Movement and Natural Communication for the Visually Impaired

The number of visually impaired people is increasing every year due to accidents, loss of vision with aging and the development of eye diseases such as glaucoma. In Japan, there are an estimated 1.64 million visually impaired people, of which 188,000 are totally blind. (*3) These visually impaired people are inconvenienced in two main situations in their daily life. First is they cannot "move freely" since they cannot check their route on a map, head to a destination, and to walk on the street while avoiding people or objects. Second is they are unable to take "natural communication" because they cannot read a person's facial expressions. Both are an essential part of our daily lives, such as going shopping, going to work or school, or traveling.

The person who attempted to use the power of technology to solve these problems is IBM Fellow (Prestigious Technological Honor) Dr. Chieko Asakawa, who lost her sight at age 14 and has a visual impairment herself. Dr. Asakwa is a researcher who has worked to improve accessibility of information for people with disabilities, including the development of the world's first website reading out software. Dr. Asakwa's policy is to "always implement technologies that she has studied to the society". Her next research theme has been "AI Suitcase," an integrated solution for supporting independent mobility using AI, which aims to improve accessibility and quality of life for the visually impaired.

"People with visual impairments cannot walk around the city alone nor enjoy themselves in peace. I want to solve these problems, but there are many hurdles to overcome to implement the technology in society. To get ove these hurdles, it is necessary for companies in different industries to integrate their latest AI and robotics technologies" said Dr. Asakwa, and the five companies that shared her thoughts established the "Consortium for Advanced Assistive Mobility Platform", a general incorporated association. And the challenge of developing an "AI suitcase" to support the mobility and communication of the visually impaired began.

*3 Source: "Social cost of visual impairment in Japan," 2006-2008, Japanese Ophthalmologist Society study group

242-02-e.jpg

The Connection Between the AI suitcase and OMRON's DNA

AI suitcase is an integrated solution consisting a set of a wearable device and a suitcase-style navigation robot that helps the visually impaired to move around the city independently. Information obtained from digital vision is supplemented by a combination of the latest AI and robotic technologies to support mobility and communication by recognizing the faces and expressions of people passing by as well as their surroundings and the presence of obstacles. By making it into a suitcase, visually impaired people can carry it around with them in their daily lives without difficulty, and the system can provide them with tailored support at all times, depending on their circumstances.

OMRON's endorsement for this project was due to the fact that it embodies its founding DNA of solving social issues through its businesses and generating new innovations through technology. From OMRON, a team led by Kazuo Yamamoto of the Image Sensing Laboratory, Research and Development Center, Technology and Intellectual Property Division, worked on the development of the system by utilizing the knowledge OMRON has accumulated in the field of image recognition.

242-03.jpg

Development Team (Kazuo Yamamoto at the Center)

Natural Communication with the Ability to Sensing the Mood

242-04-e.jpg

OMRON is responsible for developing a function that uses facial image recognition technology to support smooth communication with people. This technology is a face-sensing technology that finds people you know and determines whether it is okay to talk to them or not.

Specifically, a camera on the user's shoulder spots a person approaching users about 5 meters away. The images from this camera is enhanced using AI to recognize the face of the approaching person in real time. When a person approaches, the system uses AI-based facial recognition to determine "Who that person is?". When a person comes closer to a certain distance, the system uses an expression estimation function to acquire information such as the person's facial expression, behavior, and whether the person is looking at the user, and then determines "Whether the person is smiling or not?", "Is it okay to talk or not?". The goal is to establish a system that can give these information by notifying using voice and vibration (*4). By conveying more information to the visually impaired, the system will enable users to communicate naturally and they can "read the mood".

Nevertheless, in reality, users move around indoors and outdoors. Depending on the location, the image from camera is affected by light and brightness which makes it difficult to recognize faces and images. These technical problems need to be addressed before sensing can be made possible in a variety of everyday situations. The development team members are still working hard to clear these issues.

*4: Alps Electronics, a member of the consortium, utilizes its knowledge of tactile interfaces to guide users by transmitting information about who is presently in what state through auditory (voice) and tactile (vibration) signals.

Challenges for the Future that Continue Beyond the COVID-19 Disaster

The new coronavirus has created another challenge in the development of this "read the mood" technology. It made difficult to recognize faces and facial expressions since everyone needs to wear a mask and ensure social distance. However, new technological developments have already begun to overcome these challenges.

Yamamoto, the team leader, describes his approach to technology development as follows, "As a researcher, I want to create new technologies that are one step ahead in order to solve people's concerns. Then I want to commercialize the technologies we have developed and implement to society."

In the past, new communication tools, such as telephones and keyboards, have been created in response to the challenges and needs of people with disabilities, such as those who are blind or unable to use their hands freely. The technology developed in the AI suitcases project will also help not only the visually impaired but also those who need assistance to walk around on their own, those who need assistance with communication. It might lead to the development of robots that work collaboratively with people. The "AI Suitcase" project is also moving forward with the aim of realizing a society in which people with and without disabilities can live together while appreciating each other. We are moving forward to implementing it into a society after conducting experiments.