Limit Case

NEW GlassBench: open benchmark of silent depth failures

Robots are leaving the lab. They pick orders in warehouses, handle glassware in laboratories and install panels in the field, and every action they take begins with what their sensors report. When a sensor goes blank, the robot knows that it does not know. When a sensor returns a plausible but wrong answer, the robot acts on false geometry, and nothing in the data says anything is wrong.

These silent failures are common. On transparent objects, one widely used depth camera returned a value on almost every pixel, yet 68.6% of those values were wrong, nearly all of them measuring the surface behind the glass. Models trained to repair depth do not reliably fix this in general, and standard metrics hide the problem by averaging silent failures together with visible ones.

The data needed to fix these failures does not exist at scale. Real failures are rare, risky to stage and hard to label, because the sensor's own measurement is the thing that is wrong. Scraped images do not contain them. They have to be generated physically and calibrated against the way real sensors break.

Limit Case builds failure data and stress tests for physical AI. We find the conditions where robot perception breaks, generate them for your camera and your objects, and test every model you ship against them. We started with glass and shiny surfaces, the cases that most often break depth sensing, and we publish our benchmark openly.

- Yajan