GlassBench
An open benchmark of silent depth failures on transparent objects
Code Apache-2.0. Synthetic data, docs and figures CC BY 4.0.
Data description
GlassBench measures the depth readings that look valid but are wrong. It reports, inside the transparent-object mask, the share of pixels that are correct, missing or wrong (more than 5% off), and the share of returned pixels that are wrong. Sources: ClearGrasp (Apache-2.0) and TransCG (CC BY-NC-SA 4.0, not redistributed).
| Detail | Value |
|---|---|
| Task | Silent depth failure measurement |
| Tolerance | 5% |
| Cameras | RealSense D415, D435, L515 (ToF) |
| Scenes | ClearGrasp D415 (90), D435 test (23), D435 val (173), TransCG D435 (2,175), TransCG L515 (2,049) |
| Metrics | Correct, missing, wrong, wrong among returned |
| Repository | github.com/limit-case/glassbench |
Intended use
Depth models
Testing depth models and repair networks.
Camera checks
Checking whether a camera is safe for a given object class.
Failure detectors
Training and evaluating failure detectors.
Comparable benchmarks
Existing transparent-object datasets (ClearGrasp, TransCG) measure depth completion error; GlassBench adds the split between missing and confidently wrong readings.
Results by camera
| Camera | Correct | Missing | Wrong | Wrong among returned |
|---|---|---|---|---|
| ClearGrasp D415 (90 scenes) | 23.6 | 62.5 | 13.8 | 36.9 |
| ClearGrasp D435 test (23) | 51.6 | 32.2 | 16.2 | 23.9 |
| ClearGrasp D435 val (173) | 29.3 | 39.7 | 30.9 | 51.3 |
| TransCG D435 (2,175) | 65.3 | 26.2 | 8.6 | 11.6 |
| TransCG L515 (2,049) | 30.0 | 4.4 | 65.6 | 68.6 |
Values in percent. Opaque control (ClearGrasp) is 93.8 to 98.9% correct. Results are robust to mask erosion and per-image bias correction.
Failure statistics
At least 99.8% of wrong pixels read too far. The object-level figures are a proxy on L515, not a grasp experiment.
Models
DFNet roughly halves wrong depth on its own TransCG D435 test scenes (13.2% to 6.2%), but out of domain it raises it (ClearGrasp D415 13.8% to 39.0%) and inherits 76.6% of L515 errors. Depth Anything V2 also reads through glass.
Flagging: the ClearGrasp RGB mask network covers 91.5 to 99.3% of wrong pixels at 5%, but over-covers (1.5 to 2.6x area, IoU 36.5 to 59.7%). Failures are largely locatable from RGB.
Simulation
Blender stereo (refracted projector plus SGBM) and approximate ToF reproduce the too-far direction. After held-out calibration (at most 3 parameters, protocol fixed in advance), ToF loss falls from 43.3 to 7.4 percentage points. Stereo D415 falls from 97.2 to 38.4 and D435 test from 88.9 to 54.8, so stereo is still too often confidently wrong.
Caveats: the held-out sets were not blind and are small.
Collection method
We measure real sensor output against ground-truth depth, restrict to the transparent-object mask, and classify each pixel as correct, missing or wrong. All numbers in the repo are generated from CSVs by scripts. A validation script checks the release.
Metadata
data/ 21 CSVs, stats JSON, schemas, configs
docs/
examples/ 9-scene synthetic sample plus loader
tools/ loaders, checks, evaluation, flagging,
simulation, calibration, figures
CHANGELOG.md
CITATION.cff
DATASHEET.md
LICENSE.txtCitation
Limit Case (2026). GlassBench: an open benchmark of silent depth failures on transparent objects. https://limitcase.tech/glassbench
@misc{limitcase2026glassbench,
author = {{Limit Case}},
title = {GlassBench: an open benchmark of silent depth failures on transparent objects},
year = {2026},
url = {https://limitcase.tech/glassbench}
}Technical report: coming soon
Access
- 01
Request
Open the GitHub issue form. Tell us your camera and failing objects.
- 02
Review
We read every request.
- 03
Delivery
A free test batch, target 48 hours, for early requests.
Permitted
- Using the code and our synthetic data commercially or in research.
- Citing and building on the benchmark.
Not permitted
- Using TransCG-derived data commercially (it is CC BY-NC-SA 4.0).
- Redistributing third-party datasets.