Computational approach could continually teach robots new skills via dialogue

An example run of our framework in the user study where a user asks a robot to make sandwich, but the robot does not know how to cut cheese so it asks for the users help with language and after the user teaches this skill the robot stores this skill and can use it forever to make a similar sandwich on its own. This work is a path toward a robot that can continue to learn with human feedback on real-world chores. Credit: arXiv (2024). DOI: 10.48550/arxiv.2409.03166

···

While roboticists have introduced increasingly sophisticated robotic systems over the past decades, most of the solutions introduced so far are pre-programmed and trained to tackle specific tasks. The ability to continuously teach robots new skills while interacting with them could be highly beneficial and could facilitate their widespread use.

Researchers at Arizona State University (ASU) recently developed a new computational approach that could allow users to continually train robots on new tasks via dialogue-based interactions. This approach, introduced in a paper posted to the arXiv preprint server, was initially used to teach a robotic manipulator how to successfully prepare a cold sandwich.

"Our goal is to contribute to the deployment of robots in people's homes that can learn to cook cold meals," Nakul Gopalan, supervising author for the paper, told Tech Xplore. "We want this from a user perspective where we understand what behaviors people need from a household robot.

"This user perspective has led us to using language and dialogue when communicating with robots. Unfortunately, these robots might not come knowing everything, like how to cook pasta for you."

The key objective of the recent work by Gopalan and his colleagues was to devise a method that would allow robots to rapidly acquire previously unknown skills or behaviors from human agents.

In a previous paper, presented at the AAAI Conference on Artificial Intelligence, the team focused on teaching robots to complete visual tasks via dialogue-based interactions. Their new study builds on this previous effort, introducing a more comprehensive method for dialogue-based robot training.

"Our scope of this work is to improve the applicability of robots by allowing users to personalize their robots," Weiwei Gu, co-author of the paper, told Tech Xplore. "As robots need to complete different tasks for different users, and completing these tasks requires different skills, it is impossible for the manufacturers to pre-train robots with all the skills that they need for all these scenarios. Therefore, robots need to obtain these skills and task relevant knowledge from the users."

To ensure that a robot can effectively acquire new skills from users, the team had to overcome various challenges. First, they had to ensure that human users were engaged while teaching a robot and that the robot communicated any doubts or requested additional information in ways that non-expert users could understand.

"Second, the robot needs to capture the knowledge from only a few interactions with the users, as the users can't be stuck with the robot for an infinite amount of time," said Gu. "Last, the robot should not forget any pre-existing knowledge despite obtaining new knowledge."

Gopalan, Gu and their colleagues Suresh Kondepudi and Lixiao Huang set out to collectively address all these requirements of continual learning. Their proposed interactive continual learning system tackles these three sub-tasks via three distinct components.

A user teaching the robot a skill by holding the arm. Credit: Gu et al.

"First, a large language model (LLM)- based dialog system asks questions to users to acquire any knowledge it might not have or continue interacting with people," explained Gopalan. "However, how does the robot know that it does not know something?

"To address this problem, we trained a second component on a library of robot skills and learned their mappings to language commands. If a skill requested is not close to language the robot already knows, it asks for a demonstration."

The team's newly developed system also includes a mechanism that allows robots to understand when humans are demonstrating how to complete a task. If the demonstrations provided were insufficient and they did not reliably acquire a skill yet, the module allows robots to ask for additional ones.

"We jointly used skill representations and language representations to model the robots' knowledge of a skill," said Gu. "When the robot needs to perform a skill, it first estimates whether it possesses the capability to directly perform the skill by comparing the language representations of the skill and that of all the skills the robot possesses.

"The robot directly performs the skill if it is confident that it can do so. Otherwise, it asks the user to demonstrate the skill by performing the skill themselves in front of the robots."

Essentially, after a robot observes a user completing a specific task, the team's system determines it already possesses the skills necessary to complete it, based on the visual information gathered.

If the system predicts that the robot has not yet acquired the new skill, the robot will ask the user to delineate the associated robot trajectories using a remote control, so that it can add these to its skill library and complete the same task independently in the future.

"We connect these representations of skills with an LLM to allow the robot to express its doubts, so that even non-expert users can understand the requirements of the robot and help accordingly," said Gu.

The system's second module is based on pre-trained and fine-tuned action chunking transformers (ACT) with low-rank adaptation (LoRA). Finally, the team developed a continual learning module that allows a robot to continuously add new skills to its skill library.

"After the robot is pre-trained with certain pre-selected skills, the majority weights of the neural-network are fixed, and only a small portion of the weights introduced by the Low-Rank Adaptation is used to learn novel skills for the robots," said Gu. "We found that our algorithm was able to learn novel skills efficiently without catastrophically forgetting any pre-existing skill."

The researchers evaluated their proposed closed loop skill learning system in a series of real-world tests, applying it to a Franka FR3 robotic manipulator. This robot interacted with eight human users and gradually learned to tackle a simple everyday task, namely making a sandwich.

The robot after it completed the entire sequence of tasks and made a sandwich. Credit: Gu et al.

"The fact that we can demonstrate a closed loop skill training approach with dialog with real users is impressive on its own," said Gopalan. "We show that the robot can make sandwiches taught by users that came to our lab."

The initial results gathered by the researchers were highly promising, as the ACT-LORA component was found to acquire new fine-tuned skills with 100% accuracy after only five human demonstrations. In addition, the model retained an accuracy of 74.75% on pre-trained skills, outperforming other similar models.

"We are very excited that the robot system we designed was able to function with real users as it shows a promising future for real robot applications for this work," said Gu. "However, we do find room to improve the effectiveness of the communication of such a system."

Although the newly developed learning system yielded good results in the team's experiments, it also has some limitations. For instance, the team found that it could not support turn-taking between robots and human users, thus it relied on the researchers to elucidate whose turn it was to tackle the task at hand.

"While our findings were exciting for us, we also observed that the robot takes time to learn and this can be irritating to users," said Gopalan. "We still have to find mechanisms to make this process faster, which is a core machine learning problem that we intend to solve next.

"We want this work to get in people's homes for real experiments, so we know where the challenges in using robots in a home care situation exist."

The system developed by Gu, Gopalan and his colleagues could soon be improved further and tested on a wider range of cooking tasks. The researchers are now working on solving the turn-taking issues they observed and extending the set of meals that users can teach robots to cook. They also plan to conduct further experiments involving a larger group of human participants.

"The turn-taking problem is an interesting problem in natural interactions," added Gu. "This research problem also has strong application implications on interactive household robots.

"In addition to addressing this problem, we are interested in scaling up the size of this work by introducing more different tasks and experimenting with our system with users from real-world demographics."