gallery unit · data core
Silent data-regression guardrail
The robot data looks fine and is wrong. A sensor axis quietly changed scale overnight, and every schema check you have still passes it into training.
scenario sandbox
You are the data engineer. This morning's load is 1133 rows of the lerobot/pusht robot dataset, and it may carry one of these corruptions. The real guardrail runs in your browser, computed live, not canned.
inject a corruption
clean pass
The data as recorded passes both checks. Nothing fired, which is the point: zero false positives on clean data.
no violations, clean data passes
passed, clean data is valid
Baseline: the data as recorded. Nothing should fire. Same detectors and thresholds as the Python harness; verified to reproduce its 6/6-vs-1/6 verdicts on this sample. The cited table below is the measured 50-episode run.
what you are looking at
01 · data in
A sample of lerobot/pusht, a real robot dataset, loads in your browser. The corruption you pick is injected into a copy of it.
02 · check runs
The guardrail compares the load to a profile fitted on clean data: array dims, value stats, frame timing, episode lengths. A standard schema contract runs beside it.
03 · verdict out
Each check reports flagged or passed. The banner says in plain words which one saw the break.
Go deeper
Watch the walkthrough (45 s, silent)
Three corruptions injected, one clean pass, verdicts live · recorded from this page, unedited.
The measured result: 6/6 vs 1/6
The data looks fine and is wrong. A schema contract passes it through; this catches it.
On lerobot/pusht
6 silent corruptions + 1 loud · zero false positives · 1.8 ms/check
| Silent corruption | Guardrail | Type/presence contract |
|---|---|---|
| Dropped sensor channel | caught | missed |
| Unit / scale flip | caught | missed |
| Frame-rate drift | caught | missed |
| Clock misalignment | caught | missed |
| Episode truncation | caught | missed |
| Schema rename(the loud one) | caught | caught |
| Total | 6 / 6 | 1 / 6 |
Both catch the loud schema rename. The guardrail catches all five silent regressions the contract sails past. Numbers written by the run, not by hand.
Where the problem statement comes from
“Implement observability, validation, and guardrails to prevent silent data regressions.”
Physical Intelligence · ML Infra Engineer (Data Systems) · public job posting
An independent prototype inspired by a publicly stated problem. Not affiliated with, or a replica of, any proprietary system; the title names the problem, never a company’s internals.
The full mechanism: why a column validator misses it
A standard data contract validates columns: this one exists, has this type, isn’t null. That is the right shape for loud regressions and the wrong shape for silent ones, for two reasons.
- It can’t see inside array columns. A robot state is
[x, y]in one cell. Drop a dimension or rescale an axis and the column is unchanged, still a non-null object column. - It can’t see across the episode structure. Truncation, frame-rate drift, and clock misalignment are properties of sequences. A row-wise, column-wise validator has no concept of an episode.
And tightening it makes it worse: auto-inferring strict value ranges makes the validator false-positive on clean unseen data. A control that rejects good data isn’t detecting regressions, it’s just brittle. So the guardrail profiles the structure a column check can’t: array dimensionality, per-dimension stats, inter-frame period, per-episode start, and the episode-length floor.
clean baseline ──▶ profile ──▶ check(candidate)
• array dims → dropped channel
• per-dim value stats → unit / scale flip
• inter-frame period → frame-rate drift
• per-episode start → clock misalignment
• episode-length floor → truncation
• column presence → schema rename (loud)Evidence + how to reproduce
A controlled A/B against a real off-the-shelf validator (pandera), not a strawman. pandera’s strict statistical config was disqualified for false positives on clean holdout; that’s recorded, not hidden. Reproduce it yourself:
make setup && make test && make run # $0, laptop, no GPU
What it doesn't do
- A batch guardrail on a sampled subset, not a streaming petabyte system.
- Detects structural / distributional regressions, not semantic label errors.
- Mild truncation of a long episode is a known blind spot; the claim uses a severe (≤30%) truncation.
- Control is pandera standing in for a Great Expectations auto-profile (GE 1.x removed that profiler).