VideoManip Converts Videos of People and Objects Interacting Into Training Data

The Breakdown: 

• VideoManip teaches robots manipulation skills using videos of people interacting with objects.
• It reconstructs movements and estimates how people make contact with objects.
• The system helps robots learn new skills without time-consuming, human-operated demonstrations.

* * * 

Researchers in Carnegie Mellon University’s School of Computer Science are developing a new way for robots to learn everyday tasks like grasping a coffee mug or opening a drawer by watching videos of people interacting with objects. Their system, VideoManip, converts these videos into training data that can be used to teach robots dexterous manipulation skills.

VideoManip could eliminate the need for robot demonstrations or specialized motion-capture systems, making it easier for robots to learn how to use everyday objects.

“Humans have designed the world for human hands,” said Jeffrey Ichnowski, an assistant professor in the Robotics Institute (RI). “The tools, devices and objects around us are meant to be operated with human hands. A common challenge in robotics is determining how robots can interact with those same objects the way people do.”

VideoManip addresses one of the field’s biggest bottlenecks: data collection.

Teaching robots dexterous manipulation tasks requires large amounts of training data. Traditionally, collecting that data has involved specialized equipment, wearable sensors and hours of teleoperated demonstrations.

“While AI systems like ChatGPT can learn from massive amounts of internet data, robot learning — teaching robots how to physically interact with the world — has struggled to scale in the same way,” said Hongyi Chen, an RI Ph.D. student and lead researcher on the VideoManip team. “Collecting examples of people grasping, moving and manipulating objects is much more difficult than gathering text or images from the internet.”

VideoManip doesn’t rely on specialized equipment or human-operated robot demonstrations for training. Instead, it can use any video that focuses on a person using or manipulating an object. The system analyzes a video, reconstructs 3D movements of a person’s hands and the object, and estimates how the person makes contact with that object. It then translates those actions into movements a robotic hand can perform. The researchers found that robotic platforms trained using VideoManip could perform a variety of real-world manipulation tasks using a multifinger robotic hand.

The system can also create many additional training examples from a single video using computer simulation. By generating different versions of the same task, VideoManip gives robots more opportunities to practice and learn.

“We’re trying to enable robots to learn from videos that are readily available on the internet,” said Zackory Erickson, an assistant professor in the RI. “When we can start leveraging that, we have a clear way forward to solving the lack-of-data problem for robot learning.”

The project reflects a broader shift occurring across robotics, where researchers are increasingly exploring internet-scale datasets and foundation-model approaches to robot learning. By transforming ordinary human videos into robot training data, VideoManip offers a path toward teaching robots new skills simply by watching people perform them.

Along with Ichnowski, Chen and Erickson, the VideoManip team included Tony Dong, an SCS undergraduate student; RI master’s students Tiancheng Wu and Yash Jangir; RI Ph.D. student Yaru Niu; and RI Ph.D. graduate Homanga Bharadwaj. The RI researchers also collaborated with Liquan Wang, a Ph.D. student at the Georgia Institute of Technology, and Yufei Ye, a researcher at Stanford University.    

To learn more about VideoManip, visit the project website.

For More Information: Aaron Aupperlee | 412-268-9068 | aaupperlee@cmu.edu

Share This Story!