Artificial Intelligence breathes life into "digital sardines" - Learning from nature and giving back to global society –

Blog post   •   Apr 12, 2016 09:00 UTC

What you see here is a video image of some 15,000 sardines swimming in a large group in the boundless expanse of the ocean. Moving to the right or left in a well-coordinated manner, sardines avoid bumping into each other, with the shoal constantly changing its shape. It's hard to look at this image and not be impressed by the beautiful movements of the sardines.

At a first glance, what is shown in the video can be easily mistaken for a real shoal of sardines. But if you take a closer look, you will notice that it's not a recording of real sardines, but a computer graphics simulation.

You might think that the advancement of digital technology makes it easy to create any kind of movements with computer graphics. But in fact, moving some 15,000 sardines in a harmonious manner as seen in the natural world is not an easy task at all.

Behind this beautiful image is a computer with the latest GPU working at full capacity to process a huge amount of data. Embedded with AI technology based on precise calculations, the CG sardines move in a well-coordinated manner.

What makes this possible is OMRON's machine learning technology, which enables a computer to learn in a similar fashion as humans do naturally in the real world. This involves extracting rules or criteria for decision-making from among large quantities of data, recognizing patterns, and making predictions about new data according to these patterns. OMRON developed this machine learning technology through its development of social infrastructure systems and manufacturing automation technology. Another key technology that makes precisely coordinated movements possible is OMRON's vision sensing technology for 3D measurement and other forms of sensing.
OMRON first unveiled its "digital sardine" technology in 2013 in China. The shoal of "digital sardines" swimming exactly like real ones amazed people in China when they saw the image for the first time.

Members of the "digital sardines" development team

The technology was developed by a team comprising senior researcher Yasuyo Kotake and three other researchers—Tadashi Hyuga, Kennosuke Hayashi, and To Sho—and the team leader, Masato Kawade. With a background in brain science and systems neuroscience, Kotake is devoted to the development of new solutions to enable harmonious interaction between people and machines by drawing on image processing and biosignal processing technologies. Hyuga's specialty includes a wide range of vision sensing technologies, from character recognition to detection and recognition of a person's physical condition. Hayashi specializes in behavior recognition and interaction of people based on 3D measurement. Sho is adept at manipulating the space of digital rays through the employment of optical measurement and computer graphics technologies. Uniting this group of members with different areas of expertise is Masato Kawade, who leads the development of technologies from future-oriented and global perspectives. The "digital sardines" are the result of combining a diverse range of technologies reflecting the strengths of each member.

But it's fair to ask, "Why digital sardines?"

The answer is this: OMRON has been committed to addressing technological challenges to realize the autonomy of individuals, mutual coordination, and total harmonization to offer optimally balanced solutions that are adaptable to changes in the social environment. OMRON's goal is to build a society in which individuals with their own goals and values strive to solve issues while mutually detecting and coordinating the intentions of one another. By so doing, optimal harmonization is achieved for large-scale systems to fulfill unified goals and values.

When the team observed the way that sardines live together as one big group based on unspoken rules to protect themselves from underwater predators, they realized that the technology pursued by OMRON would also require following the laws of nature.

This inspired Kotake and other members of the team to thoroughly study the behavior of sardines.

The purpose of life leads to total harmonization

A sardine run—the phenomenon of thousands of sardines swimming in a big shoal

Occasionally, a sardine run may be seen with more than 50,000 sardines migrating in the ocean. The sardines group together to form a huge ball, and make unique movements to alter the shape of the ball from minute to minute to scare away enemies. Because sardines have many predators, a communal lifestyle is indispensable for this species. A smaller group of sardines, or those separated from the group, would easily become the prey of larger fish. This is why sardines instinctively have a habit of swimming close together, forming large, ball-shaped groups.

But sardines group together not just to escape attack from predatory fish. Sardines have a keen sense of smell, so if one or two of them detect food, it is transferred to others in the group so that they all move toward the food in a big group. Sardines inform others of the presence of predators or food and this is how they survive.

At first, Kotake and the others thought they could reproduce a school of sardines using a CG technique, setting parameters to control CG-created sardines that would make them move to avoid colliding with each other. Parameters included the distance from surrounding entities (mates, food, and predators), and the speed and direction of swimming. But their CG sardines didn't move as intended. They seemed to move randomly without intention.

The team surmised that the cause of the failure may be that they didn't give each sardine a purpose of life—living communally in order to survive.

The team went back to the drawing board.

Sardines leading a communal lifestyle are said to have the following three behavioral patterns:
1. Escaping when an enemy approaches
2. Swimming close together to form a big ball
3. Finding and consuming food

Sardines recognize ever-changing marine conditions, and move while changing their priorities for these three patterns. An AI algorithm to simulate this behavior had to be developed. CG sardines stopped moving when there were too many parameters to control their movements. If a sardine's unique personality was too strong, it went out of control so total optimization could not be achieved. The team underwent repeated trial-and-error adjustments. To faithfully reproduce the movement of real sardines, the team compared the movement of CG sardines with that of real ones. They did this over and over and made repeated improvements.

This process finally resulted in a community of "digital sardines" in which each sardine can think on its own and coordinate its behavior with that of others in the group. In this way, each sardine worked toward a single unified purpose, while maintaining its unique potential. Kotani provides her perspective:

"In the past, I have developed image processing techniques for detecting the flow of people or the characteristics of goods being conveyed on a production line. To allow a machine to interpret the behaviors of a person or those of other machines, the machine must be able to think on its own to discover optimal parameters, and take action according to the parameter data. To allow each sardine to individually change numerous parameters at a high speed to derive an optimal solution for the whole has things in common with how image processing works."

The technique used to come up with an optimal solution for the "whole" is similar to the technology that allows a car to detect traffic conditions, come up with the best route, and avoid traffic congestion. This is not only optimal for the individual driver, but it also reduces traffic congestion for the "whole" of the drivers on the road.

There are several other technologies that have things in common with this technique. One example is technology to prevent accidents at intersections, by allowing pedestrians, bicycles, and cars to warn each other of their relative positions. Technology to monitor the electricity consumption of each household in a town will also help the smart grid to supply power generated by solar panels and stored in battery units to households when necessary. All these technologies now under development at OMRON will enable total optimization, which in turn will ensure safe and convenient living for all.

"I want to continue developing new technologies to allow machines to correctly interpret the behavior or intentions of people or other machines. This can provide many benefits to our society," Kotake concluded.

Each one of the sardines leading a communal lifestyle can be related to each one of us as humans. That is why we should learn from nature and give our knowledge back to global society in the form of innovative technology. OMRON will continue to help create an ideal society for the future—a society in which total optimization and harmonious balance will be enjoyed among people, machines, and the earth.

A conceptual illustration of a future street intersection where pedestrians, cars, and bicycles exchange positional information with each other