Physics engines like MuJoCo, Isaac Gym and Genesis model rigid-body dynamics well. Rendering engines like Unreal and Blender give you photoreal digital twins. But both share a blind spot: they do not simulate how agents actually see. A real single-photon LiDAR pointed at a retroreflective sign reports a phantom obstacle in front of it. A real LWIR camera shows the lingering thermal footprint of a warm handprint. A long exposure blurs whatever moved. These are not exotic edge cases — they are the operating conditions that decide whether a deployed robot succeeds or fails.
VisionSIM (UW–Madison, CMU, Portland State) makes sensor realism a first-class concern, modelling the raw physical measurement process rather than the scene geometry around it. It supports conventional and stereo cameras, passive SPAD cameras, dToF and iToF LiDAR, event cameras, thermal cameras and IMUs — each with fine-grained control over physical parameters. The point is to reproduce modality-specific failure modes: pileup and glare for LiDAR, phase wrapping and saturation for iToF, refractory drops for event cameras, history-dependent emission for thermal.
The Numbers That Matter
The showcase is a digital twin of a commercial single-photon LiDAR (Adaps Photonics ADS6311). Because the dToF model is differentiable, sensor parameters are recovered from real captures by gradient-based fitting — including the impulse response function, whose dead time reproduces pileup and range-walk error. Modelling IRF, pileup and glare correctly lowers the twin’s depth RMSE by ~1.53 metres.
Dataset migration is the second lever: existing assets can be uplifted to modalities they were never captured in — a XVFI high-speed video into a single-photon camera, a Genesis humanoid render into an event stream, a CARLA driving sequence into raw LiDAR transients. One consequence is instructive: lifting a driving dataset into time-resolved single-photon measurements makes pedestrians invisible at the sensor level, because lateral motion smears the transient histograms into depth errors beyond 3 metres.
The third use case is buying decisions before hardware exists. A system architect replays one scene through two candidate suites — conventional camera + event camera versus SPAD camera + IMU — through an off-the-shelf detector. The first fails; the second yields 90% confidence.
Fidelity Checks
Sim-to-real gaps are reported per modality rather than asserted. A simulated mug of hot tea tracks a real FLIR Boson capture to within 7.6% of the heating dynamic range over four seconds. An emulated iToF sensor hits a mean phase error of 0.13 radians against a real Chronoptics capture, i.e. roughly 5.3 cm of depth error at 40 MHz. Event counts emulated from RGB video differ from a real Prophesee EVK4 recording by 1.5% (positive) and 5.8% (negative). Emulated SPAD photon-detection statistics reach 0.221 against a measured 0.229.
The released dataset covers 100 indoor scenes with a predefined 24-second trajectory each, rendered at 100 fps in HDR at 1000×1000 as a stereo pair — 480,000 synchronized ground-truth views with depth, normals, optical and scene flow, segmentation and temperature — then emulated into a 16 kHz SPAD camera, 4 kHz event camera, 25 fps camera, dToF/iToF LiDAR, thermal sensor, 250 Hz IMU and stereo rig.
The honest caveat: the case studies are open-loop and focus on sensor fidelity, not closed-loop control. And simulated-to-simulated comparisons inherit the biases of the reconstruction and detection models used zero-shot on top.
The takeaway: if your robot stack already works in sim and dies in the real world, the gap is more likely in the sensor model than in the policy.
Paper: arXiv:2611.08368