Low-latency time-of-flight non-line-of-sight imaging
at 5 frames per second



Ji Hyun Nam
Eric Brandt
Sebastian Bauer
Xiaochun Liu
Marco Renna

Alberto Tosi
Eftychios Sifakis
Andreas Velten


Computational Optics Group, University of Wisconsin - Madison


Paper
Supplementary Materials
Code
Data
Bibtex
Youtube Video
Media

Abstract

Non-Line-Of-Sight (NLOS) imaging aims at recovering the 3D geometry of objects that are hidden from the direct line of sight. One major challenge with this technique is the weak available multibounce signal limiting scene size, capture speed, and reconstruction quality.

To overcome this obstacle, we introduce a multipixel time-of-flight non-line-of-sight imaging method combining specifically designed array detectors with a novel fast reconstruction algorithm that for the first time captures and reconstructs live real-time videos of non-line-of-sight scenes with natural non-retroreflective objects. We develop a model of the signal-to-noise-ratio of non-line-of-sight imaging and use it to devise a method that reconstructs the scene such that signal-to-noise-ratio, motion blur, angular resolution, and depth resolution are all independent of scene depth suggesting that reconstruction of very large scenes may be possible.



Real-time NLOS Imaging
DEMO Video



Real-time NLOS setup. The system captures signal from the hidden scene and reconstructs the scene and displays the result on the monitor in real-time.

Person moves target (digit 4), performs some exercise, and throws a toy. The system successfully reconstructs the scene.

Room light on. We can even reconstruct a mirror! Person is wearing a gray hoodie which you can buy in "GAP" stores.
Watch full video here: Youtube Video

How it works?



Real-time NLOS virtual image processing pipeline. The imaging system sends the virtual Phasor Field (PF) signal to the visible wall and captures the signal returning from the hidden scene back to the wall. The massive raw photon stream is recorded by the SPAD array. Raw photons from all channels are virtually remapped into a complete aperture. Then the remapped data is transformed into the frequency domain (Fourier Domain Histogram, FDH) and propagated by the fast Rayleigh Sommerfeld Diffraction (RSD) algorithm. Lastly, temporal frame averaging yields constant SNR throughout the entire reconstructed volume, and the result is displayed.




Real-time DEMO Video
The person takes out 4 letters("N", "L", "O", "S") from the box.



Comparison with other methods


To demonstrate the effectiveness of our method, we provide a baseline comparison with other methods. Besides the two 16x1 non-confocal SPAD arrays, our imaging system has a single pixel gated SPAD that can be used for confocal data acquisition. We collect data using confocal and non-confocal system with different exposure time and compare reconstruction quality.

The figure shows the comparison which is split into two parts, non-confocal and confocal data. The non-confocal data are used for direct reconstruction with the proposed method, and are also approximately converted to confocal data with the method described in Lindell et al.[1] , as previously also demonstrated in [2]. Then, the LCT [3] ("approximate LCT") and FK migration [1] ("approximate FK") are applied to this approximately confocal data. The reconstruction results of approximate LCT and approximate FK look blurry, have a hazy background, and look noisy at short exposure times. Our method successfully reconstructs the scene even with a short exposure time of 0.2s. After 1-4 seconds of exposure the reconstruction quality doesn't change noticeably, which suggests our proposed method doesn't require long exposure times. Previously Liu et al. [4] demonstrated that a Phasor Field reconstruction stabilizes at a certain exposure level and adding further exposure time does not affect the reconstruction.

Using the confocal measurements we reconstruct the scene using the confocal Phasor Field reconstruction version[2], LCT[3] and FK[1]. The confocal acquisition method in general suffers from noise with short exposure times and requires a long exposure time to achieve high quality reconstruction of diffuse targets. Therefore, often times confocal methods use retroreflective materials to increase the signal. Since the confocal measurement requires a dense scan of the relay wall, a confocal scan at 0.2 seconds was not possible with our galvo mirrors. Note that the times reported here are in line with what was reported in the original papers.


[1] David B. Lindell, Gordon Wetzstein, and Matthew O’Toole. Wave-based non-line-of-sight imaging using fast f-k migration.ACM Trans. Graph. (SIGGRAPH), 38(4):116, 2019.
[2] Xiaochun Liu, Sebastian Bauer, and Andreas Velten. Phasor field diffraction based reconstruc-551tion for fast non-line-of-sight imaging systems.Nature Communications, 11(1):1645, April 2020.
[3] Matthew O’Toole, David B Lindell, and Gordon Wetzstein. Confocal non-line-of-sight imaging based on the light-cone transform.Nature, 555(7696):338, 2018.
[4] Xiaochun Liu, Ib ́on Guill ́en, Marco La Manna, Ji Hyun Nam, Syed Azer Reza, Toan Huu Le, Adrian Jarabo, Diego Gutierrez, and Andreas Velten. Non-line-of-sight imaging using phasor-field virtual wave optics.Nature, 572(7771):620–623, 2019





Acknowledgements

This work was funded by DARPA through the DARPA REVEAL project (HR0011-16-C-0025), National Science Foundation grants NSF IIS-2008584, CCF-1812944, and IIS-1763638 and a grant from UW--Madison's Draper Technology Innovation Fund.