Delivering technology that helps individuals grow. ~ From beginner to advanced. A robot that plays table tennis at your level~

In the past the main design criteria for industrial robots was to improve productivity and to work at a higher efficiency than humans.

Now however, the relationship between humans and robots is not as simple as humans giving orders and robots doing the work. People have come to expect a better level of interaction between themselves and machines such as communication robots, which can sometimes be seen in customer service areas, and the AI personal assistants included in smartphones.

What does the future hold for human-machine cooperation? OMRON believes that people can grow as an individual through the support of technology via the concept of "harmony between people and machines". A primary example being the table tennis playing robot "FORPHEUS".

Since its inception in 2014, FORPHEUS has allowed people all over the world to experience "harmony between people and machines," while continually evolving through technological development and feedback from those who have experienced it at exhibitions, etc. The 4th generation of FORPHEUS, developed in 2017, has expanded the range of potential candidates it can maintain a rally with. In this article we will follow the development of the robot and the challenges faced along the way.

Responding to the emotions of those around it, increasing enjoyment for all.

Mr. Koizumi, development leader for the 4th generation FORPHEUS.

In order to make a table tennis rally enjoyable there are certain conditions which must be met such as being able to serve, responding to an opponent's return ball, especially a smash, and to maintain the motivation to keep the rally going. For this generation we aimed to go beyond these conditions, to evolve to a point where a greater variety of people could experience a rally with the robot.

"We have asked many people to try to maintain a rally with the robot, but beginners often fail at the very first hurdle, the serve, and so are unable to even start. In addition, I felt that higher level players could not enjoy the experience through a simple rally." Said the development team leader Masayuki Koizumi on previous iterations.

To address these shortcomings three functions were chosen as development goals, the ability to serve, the ability to predict a smash and a sound function.

"We identified that about 30% of beginners failed at the serve so we thought we would try to switch the responsibility over to the robot. We also knew that advanced players tended to try to challenge the robot with a smash. Wouldn't it be surprising if the robot could predict that a smash was coming? If we added an audio component to the rally, would it increase the realism, allowing people to enjoy the experience more? These were the thoughts I had when observing people playing against FORPHEUS.

Observe and mimic, paying attention to the human body.

We wanted everyone to be able to improve while playing against the robot, regardless of starting ability. This presented numerous challenges during development.

Tossing the ball up and predicting a smash are relatively simple tasks for a human however recreating the motion in a robot represents a significant challenge. Rather than start from scratch, we focused on the movement of the human body as a base.

"We repeatedly watched slow motion replays of humans tossing the ball up into the air. We discovered that this one movement was actually made up of a series of smaller actions: lowering the fingertips, a short drop, twisting the writs and then releasing the ball." Said Daichi Ueki, who oversaw developing the robot's serve function. Through constant repetition and refinement, we could get the robot's hand to toss in a similar way to a human.

Getting ready to serve.

Hiroshi Maruyama and Kyohei Asai of the development team worked on smash prediction.

Originally, FORPHEUS had trouble returning balls that had been hit hard. It is difficult to detect a ball right after a smash as it is moving significantly faster than a normal hit. Rather than trying to detect the ball directly after a smash, the two developers instead decided to focus on the openness of the human body when performing the technique.

"We observed "tells" in many people before they hit a smash, wiggling of the hands while moving the shoulders and an "opening up" of the body being two of the main ones. We decided to train the camera to look for these signs that a smash was coming." Said Maruyama.

Mr. Asai implemented the response for when a smash was predicted.

"Once we could predict what was coming we had to work out how to deal with it. We trained FORPHEUS to reproduce the movement a real table tennis player would take to counter the shot, i.e., hold the racket in the predicted path of the ball and then block and return it."

Additionally, Nakayama Masamune, a first-year employee, improved the motion control system to deal with slower shots which traditionally had troubled FORPHEUS, and improved the camera system to check the position of the returning ball. This refinement allowed FORPHEUS to deal with a lot of the slower shots that it had previously had issues with.

Mr. Asai (left) and Mr. Nakayama (right) test the smash prediction function.

The final addition to the robot was the sound system which plays music during the rally adjusting the sound based on the speed and position of the returning ball. This helps the player to build up a rhythm during the rally. Additionally, FORPHEUS will make regular comments about how it's opponent is doing in the rally.

As a way to keep the rally fun, Mr. Koizumi focused on the relationship between what humans hear and their concentration level. He says, "I got the idea that we could use sound to enhance the human opponent's concentration level from someone who has done advanced research in the field. Even a professional table tennis player will perform poorly if you block their ability to hear. During a real table tennis match, humans use rhythm, sounds and the voice of their opponent to aid in concentration, I thought that music could be used as a motivational tool to do the same."

As a result of this trial and error process, FORPHEUS has evolved further along the path of allowing people to experience "harmony between people and machines".

Over 15,000 visitors to CEATEC JAPAN 2017 got to observe FORPHEUS in action.

What enables OMRON to create a table tennis robot?

While FORPHEUS is a table tennis playing robot, it is made from OMRON's industrial equipment, the core technology "sensing & control + Think".

Masato Kawade, who oversees the development of FORPHEUS says, "Sensing the trajectory of the ball, controlling the robot precisely, and changing the movement of the robot according to the opponent. OMRON is able to do all of these at the same time. Every day we combine products in new and innovative ways to meet our customer's needs. For example, the technology which allows FORPHEUS to judge what type of shot is bring played against it has applications in any situation where a robot must recognize things in three dimensions, such as grabbing parts at a factory. FORPHEUS's chronological deep learning technology improves the speed with which a machine makes decisions and takes actions, such as in crisis prediction systems in automatic driving technology.

   Kawade who oversaw development

When FORPHEUS returns the ball while saying "Let's continue having fun", this is what represents OMRON's vision of "harmony between people and machines" where people can improve their abilities through the assistance of robots in everyday society.