Robotic teleoperation data startup XDOF launches with $70M in funding

Robotics training infrastructure startup XDOF said today it has raised $70 million in funding to try to solve one of the biggest challenges in artificial intelligence: teaching machines the skills they need to safely navigate and work in the real world.

The round involved a number of heavyweight venture capitalists, including Thrive Capital, Spark Capital, Andreessen Horowitz, Lux Capital and WndrCo. In addition to the money, the startup also released ABC-130K, which it says is the world’s largest open-source bimanual robot manipulation dataset. It will provide robotics researchers with access to an unprecedented amount of high-quality, freely available training data.

XDOF’s debut comes at a critical juncture, just weeks after OpenAI Group PBC announced it’s going to revive its own robotics training program that had been shut down in 2021. That move signified the growing interest in what’s known as “physical AI,” but frontier model makers face a significant challenge. While large language models can be trained on vast oceans of easily-accessible data from the internet, building intelligent robots requires much more nuanced data that captures very specific, real-world actions and interactions.

This data is so scarce that it’s essentially nonexistent. Some developers have tried to get around this problem by downloading YouTube videos or using low-quality footage captured by factory workers and so on, but this data is virtually impossible to reconcile with the complex spatial requirements of robots.

Co-founder and Chief Executive Philipp Wu told TechCrunch in an interview that he experienced this challenge himself while studying as a Ph.D. student at the University of California at Berkeley. “We didn’t have large-scale data to work with,” he explained. “There was this chicken-and-egg problem — we first needed to actually collect data before we could even ask how to train a foundation model for robotics.”

XDOF believes physical AI’s biggest hurdle is not the models that actually power the robots or the high-end chips needed for onboard processing, but the data feedback loops needed to teach robots physical interactions. That’s why the company is focused on building the highly specialized data pipelines, data collection tools and annotation systems needed to gather this essential training resource. It’s an entirely new category of infrastructure, Wu said.

The startup traces its roots back to a project called GELLO that Wu worked on alongside a number of other UC Berkeley researchers. With GELLO, they developed a low-cost teleoperation system that enables human operators to control robotic arms and perform various tasks to generate accurate training data. When he teamed up with Chief Technology Officer Fred Shentu and Chief Operating Officer Nemo Jin, Wu realized that simply creating the data itself is a poor business model, so they also decided to offer data cleaning and annotation services and develop specialized tools, creating a self-reinforcing feedback loop.

The ABC-130K dataset is meant to be a showcase of what XDOF can do. It includes 130,000 trajectories of robotic manipulation data, plus 300 hours of simulations and 100 hours of evaluations. The startup has used this dataset itself to train robots on a number of tasks that require extreme precision, such as folding T-shirts, flattening cardboard boxes and putting AirPods into their plastic cases. Although it has been operating under the radar until now, it already has about 20 active customers, including a number of frontier AI labs, and more than 60 employees.

Wu said XDOF will scale its business across a three-tier “data pyramid” that includes bespoke teleoperation data that’s collected directly from the remote operation of the specific robot being trained. The middle tier includes generalized teleoperational data, similar to what GELLO produced. Finally, it includes “egocentric” data that’s gathered by humans performing the everyday tasks that robots need to learn.

Of course, creating all of this teleoperation data is going to be a significant undertaking, which is precisely why XDOF needed money. It’s going to hire a global arm of teleoperators and data gatherers. It will even develop its own proprietary wearable sensors to ensure that whatever robots are being trained will match the hand-tracking algorithms it has developed.

Because creating this data is such a labor intensive job, XDOF believes that AI labs will be only too happy to outsource it. “You need a warehouse of hundreds of thousands of square feet with hundreds of robots,” Wu explained. “You need to maintain these robots, calibrate their physical parameters and properly train operators.”

Image: XDOF