The most insidious bugs in Physical AI are in your data pipeline

Three buckets of multimodal data processing where Physical AI needs new infrastructure.

The most insidious bugs in Physical AI aren't in your model. They're in your data pipeline.

Training data in Physical AI is rapidly growing in both size and complexity. It is no longer just time-synchronized video streams; training payloads now carry hand positions, depth, and sound.

This new variety of data, and the velocity at which it is collected, raise problems that did not previously exist, because nothing before Physical AI produced data quite like this.

We suspect these problems are common across Physical AI teams, and we are still early in solving them. We hope to share our notes in the open and invite discourse around this.

Three buckets of multimodal data processing at scale where we see the need for new solutions, with early ideas on promising directions:

Reliably annotating video data with metadata (e.g. quality scores) at scale. Egocentric data today requires running task segmentation over terabytes of episodes, filtering out idle scenes that are not productive for downstream use. Pipelines backed by guarantees that the processing logic does exactly what it claims, conventionally used in high-stakes domains like high-frequency trading, could be repurposed here.
Catching calibration drift early. Detecting when a collector is wearing a device incorrectly or a camera has blurred, and flagging it in time, not after weeks of samples are collected and have to be thrown away. Observability tools could provide continuous monitoring that catches warning signs of drift before the errors compound, similar to what Datadog does for software today.
Lossless transforms. Speeding up a video trajectory, for example, without introducing noise that quietly corrupts model evaluation. Instead of naive speed-ups, we could apply statistical methods that proxy the properties of a sped-up sample. Similar methods have already been proven for normalizing motion speed.

Running compute-intensive processing like this across terabytes of video can introduce systematic errors that stay undetected until the data reaches model development, where they surface as confusing robot performance.

Debugging robot performance is already a complex task when developing Physical AI models, with many confounding variables at play. Robust multimodal data processing infrastructure is more critical now than ever, so the data layer does not become an additional point of error.

Existing solutions lean heavily on low-cost human labor for manual annotation and review. AI tools that augment manual annotation pipelines have long existed, but we worry they will not keep up with the future demands of Physical AI.

If these challenges resonate, we'd love to hear how you're thinking about them and share what we know!