Scoring egocentric data without a robot in the loop

We get asked a lot about how to quickly evaluate whether data is productive for model training without a robot in the loop. We'll focus on egocentric video data first.

We're still piecing together the fuller picture, but thought we'd share some initial findings and catalyze discussion.

Real-world rollouts still seem to be the gold standard, but they are slower, hardware-bound, and impossible to run for every candidate dataset. Simulation-based evaluations are emerging as a faster, cheaper alternative, given the advances in sim2real.

The other confounding variable: the industry is moving fast, and the shape of the data researchers want for model training is constantly changing.

The core motivation for a quicker evaluation is experiment prioritization. Training payloads are already large and will only grow as richer sensor data enters model training, and with compute in short supply, experiments are bottlenecked by which data to run first.

Over the past year, an offline proxy strong enough to build an evaluation pipeline around has emerged from Apple, unsurprisingly built on the Vision Pro and ARKit, one of the most advanced AR stacks today.

The paper is EgoDex, and the offline evaluation for data quality works by predicting future hand trajectories. This can be done because the hand trajectories are captured by the Vision Pro's on-device SLAM when the data is collected.

A model can be trained to take an input frame and predict where the hands go over the next one to three seconds: wrist pose plus five fingertips per hand. The evaluation can then be done automatically by scoring the distance between predicted and actual trajectories.

You can refer to the paper for the full details.

Training this model still requires compute, but significantly less (8 A100s, ~72 hours per run). The total cost becomes GPU-days on more accessible hardware, with no human annotation required.

A direct correlation with downstream robot performance is not a guarantee, but this is one promising data point in an ensemble of indicators we can use as a proxy for productive model training.

Lastly, EgoDex assumes hand positions come from on-device SLAM. We could apply the same technique to ego videos that lack such annotations by using a model like HaMeR to generate them retroactively.

However, we then introduce new sources of error, including HaMeR's own limitations, which is why the emerging pattern seems to be capturing this data reliably at collection time instead, using hardware like ARKit-enabled iPhones.

If you're looking to score your egocentric data on how productive it is for robot training, reach out!