Two hours of teleop can teach a robot a new task

Two hours of high quality teleop could be enough to teach a robot a new task.

One notable open-source contribution proving this today comes from mimic-video, a video-action model that achieved bimanual manipulation with dexterous hands.

They trained the action decoders from scratch on only ~500 episodes per task (just 1h 33m of demonstrations for package sorting, 2h 14m for tape stowing).

For bimanual gripper setups, we've heard teams adapt on as little as 30 minutes of teleop data with video-action models.

The common feedback we've been getting from teams curating high-quality datasets with us is that they want models that adapt fast.

Video-action models provide a promising path toward that.

We could take a pretrained video model, fine-tune it on videos of the task environment, and train the action decoder on high quality robot data from scratch.

From our understanding, Rhoda AI is approaching this similarly with their Direct Video-Action model (DVA).

Fine-tuning on video first teaches the pretrained model how the task's environment works, so the action decoder only has to learn the actions of the task from a minimal set of high-quality teleop samples.

A video model fine-tuned faithfully enough also doubles as an RL environment, which can be used to close the gap to 90%+ task performance.

From where we see things, there is a gap in open source today: models deployment teams can build on and fully own.

We think there is a place for more common infrastructure that model development teams can build upon.

If you're a robotics company that could benefit from a model that adapts to your environment and have weights you fully own, reach out.