Why a two-hour robot dataset takes days to train on

A two-hour robot dataset sounds small, but training on it still takes days.

The reason lies in what a single training example actually amounts to when training AI models on robot data, compared to LLMs and VLMs.

For a text LLM, one training sample is a snippet of conversation, which adds up to a few hundred tokens. For a vision-language model (VLM), it's an image and a question, typically a thousand tokens or so.

A robot training sample can carry up to three camera feeds, and one training episode becomes a grid of patches covering every position in every frame, with the frame count set by the sampling rate and input config.

This is how a single video-action model's training input reaches up to 19,200 tokens, compared to a thousand tokens for VLMs.

Attention cost grows with the square of the token count.

Twenty times more tokens per example than a VLM (thousand vs. 20K) requires almost four hundred times more computation on one robot training example than on a single text-plus-image training sample.

The problem is compounded because every trick that made LLM training fast, from the kernels to the managed fine-tuning services, was built around small, uniform, text-shaped inputs.

Most of it was not built for Physical AI payloads.

Every team in robotics pays the tax of figuring out how to handle these payloads in silos, and the same optimizations get rediscovered a dozen times over by others who eventually face the same issues.

We're excited to build the core tools that address these fundamental challenges in Physical AI and to discover what that unlocks for the robotics ecosystem.

If your team could use help processing large amounts of multimodal robot data for AI training, reach out.