gallery unit · data core · bottleneck class A
Silent data-regression guardrail
“Implement observability, validation, and guardrails to prevent silent data regressions.”
Physical Intelligence · ML Infra Engineer (Data Systems) · public job posting
An independent prototype inspired by a publicly-stated problem. Not affiliated with, or a replica of, any proprietary system — the title names the problem, never a company’s internals.
The data looks fine and is wrong. A schema contract passes it through; this catches it.
On lerobot/pusht
6 silent corruptions + 1 loud · zero false positives · 1.8 ms/check
| Silent corruption | Guardrail | Type/presence contract |
|---|---|---|
| Dropped sensor channel | caught | missed |
| Unit / scale flip | caught | missed |
| Frame-rate drift | caught | missed |
| Clock misalignment | caught | missed |
| Episode truncation | caught | missed |
| Schema rename(the loud one) | caught | caught |
| Total | 6 / 6 | 1 / 6 |
Both catch the loud schema rename. The guardrail catches all five silent regressions the contract sails past. Numbers written by the run, not by hand.
run it yourself
This runs the real guardrail in your browser on 1133 rows of lerobot/pusht. Inject a corruption and watch the guardrail catch what the schema contract misses — computed live, not canned.
Baseline: the data as recorded. Nothing should fire.
no violations — clean data passes
passed — clean data is valid
Same detectors and thresholds as the Python harness; verified to reproduce its 6/6-vs-1/6 verdicts on this sample. The cited table above is the measured 50-episode run.
Why a column validator misses it
A standard data contract validates columns: this one exists, has this type, isn’t null. That is the right shape for loud regressions and the wrong shape for silent ones, for two reasons.
- It can’t see inside array columns. A robot state is
[x, y]in one cell. Drop a dimension or rescale an axis and the column is unchanged — still a non-null object column. - It can’t see across the episode structure. Truncation, frame-rate drift, and clock misalignment are properties of sequences. A row-wise, column-wise validator has no concept of an episode.
And tightening it makes it worse: auto-inferring strict value ranges makes the validator false-positive on clean unseen data. A control that rejects good data isn’t detecting regressions — it’s just brittle. So the guardrail profiles the structure a column check can’t: array dimensionality, per-dimension stats, inter-frame period, per-episode start, and the episode-length floor.
clean baseline ──▶ profile ──▶ check(candidate)
• array dims → dropped channel
• per-dim value stats → unit / scale flip
• inter-frame period → frame-rate drift
• per-episode start → clock misalignment
• episode-length floor → truncation
• column presence → schema rename (loud)Evidence
Tier 4 — a controlled A/B against a real off-the-shelf validator (pandera), not a strawman. pandera’s strict statistical config was disqualified for false-positing on clean holdout; that’s recorded, not hidden. Reproduce it yourself:
make setup && make test && make run # $0, laptop, no GPU
What it doesn’t do
- A batch guardrail on a sampled subset, not a streaming petabyte system.
- Detects structural / distributional regressions, not semantic label errors.
- Mild truncation of a long episode is a known blind spot; the claim uses a severe (≤30%) truncation.
- Control is pandera standing in for a Great Expectations auto-profile (GE 1.x removed that profiler).