CS-MUVI Video compressive sensing for spatial multiplexing cameras

1. CS-MUVIVideo compressive sensing for spatial multiplexing cameras Aswin Sankaranarayanan, ChristophStuder, Richard G. Baraniuk Rice University

2. Single pixel camera Photo-detector Digital micro-mirror device

3. Single pixel camera • Each configuration of • micro-mirrors yield ONE • compressive • measurement • Non-visible • wavelengths • Sensor material costly • in IR/UV bands • Light throughput • Half the light in the scene is directed to the photo-detector • Much higher SNR as compared to traditional sensors Photo-detector Digital micro-mirror device

4. Single pixel camera • Each configuration of micro-mirrors yield ONEcompressive measurement • staticscene assumption • Key question: • Can we ignore motion • in the scene ? Photo-detector Digital micro-mirror device

5. SPC on a time-varying scene time varying scene • Naïve approach: Collect Wmeasurements together to compute an estimate of an image what happens ? t=1 t=W measurements ? compressive recovery

6. SPC on a time-varying scene t=W (large) t=W (small) t=1 Tradeoff Temporal resolution vs. spatial resolution Large W Higher spatial resolution More motion blur Small W Less motion blur Lower spatial resolution

7. SPC on a time-varying scene sweet spot Lower spatial res. Higher temporal res. Higher spatial res. Lower temporal res.

8. Dealing with Motion • Motion information can help in obtaining better tradeoffs [Reddy et al. 2011] • State-of-the-art video compression

9. Dealing with Motion • Motion information can help in obtaining better tradeoffs [Reddy et al. 2011] • State-of-the-art video compression naïve reconstruction motion estimates

10. Key points • Motion blur and the failure of the sparsity assumption • Use least squares recovery ? • Recover scene at lower spatial resolution • Lower dimensional problem requires lesser number of measurements • Tradeoff spatial resolution for temporal resolution • Least squares and random matrices • Random matrices are ill-conditioned • Noise amplification • Hadamardmatrices • Orthogonal (no noise amplification) • Maximum light throughput • Optimalfor least squares recovery [Harwit and Sloane, 1979]

11. Hadamard + least sq. recovery Hadamard Random

12. Hadamard+ least sq. recovery

13. Designing measurement matrices • Hadamard matrices • Higher temporal resolution • Poor spatial resolution • Random matrices • Guarantees successful l1 recovery • Full spatial resolution • Can we simultaneouslyhave both properties in the same measurement matrix ?

14. Dual-scale sensing (DSS) matrices Key Idea: Constructing high-resolution measurement matrices that have good properties when downsampled 1. Start with a row of the Hadamard matrix 3. Add high-freq. component 2. Upsample

15. CS-MUVI: Algorithm outline 1.obtain measurements with DSS matrices t=t0 t=t0+W t=T t=1 t=W 2.low-resolution initial estimate 3.motion estimation 4.compressive recovery with motion constraints

16. Simulation result

17. CS-MUVI on SPC Single pixel camera setup Object InGaAs Photo-detector (Short-wave IR) SPC sampling rate: 10,000 sample/s Number of compressive measurements: M = 16,384 Recovered video: N = 128 x 128 x 61 = 61*M

18. CS-MUVI: IR spectrum initial estimate Upsampled Recovered Video Joint work with Xu and Kelly

19. CS-MUVI on SPC Naïve frame-to-frame recovery CS-MUVI Joint work with Xu and Kelly

20. CS-MUVI summary • Key ingredients • Novel Measurement matrix design • Exploiting state-of-the-art motion model • One of first practicalvideorecovery algorithm for the SMC dsp.rice.edu

21. CS-MUVI summary • Limitations • Need a priori knowledge of object speed • Motion at low-resolution • Robustness to errors in motion estimates • Future work • Dual-scale to multi-scale matrix constructions • Multi-frame optical flow • Online recovery algorithms dsp.rice.edu