Real-Time 3D Model Acquisition

1. Real-Time 3D Model Acquisition Princeton University Stanford University Hunter College Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy Ilya Korsunsky

2. 3D Scanning

3. Possible Research Goals • Low noise • Guaranteed high accuracy • High speed • Low cost • Automatic operation • No holes

4. 3D Model Acquisition Pipeline 3D Scanner

5. View Planning 3D Model Acquisition Pipeline 3D Scanner

6. View Planning Alignment 3D Model Acquisition Pipeline 3D Scanner

7. View Planning Alignment Merging 3D Model Acquisition Pipeline 3D Scanner

8. View Planning Alignment Done? Merging 3D Model Acquisition Pipeline 3D Scanner

9. View Planning Alignment Done? Merging Display 3D Model Acquisition Pipeline 3D Scanner

10. 3D Model Acquisition Difficulties • Much (often most) time spent on “last 20%” • Merging, Displaying, View Planning • Pipeline not optimized for hole-filling • Not sufficient just to speed up scanner –must design pipeline for fast feedback

11. Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Alignment Human Done? Merging Display

12. Alignment Merging Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Challenge: Real Time Done? Display

13. Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Alignment Part I: Structured-LightTriangulation Done? Merging Display

14. Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Alignment Part II: Fast ICP Done? Merging Display

15. Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Alignment Part III: Voxel Grid Done? Merging Display

16. Laser Camera Triangulation • Project laser stripe onto object Object

17. Camera Triangulation • Depth from ray-plane triangulation Object Laser (x,y)

18. Triangulation • Faster acquisition: project multiple stripes • Correspondence problem: which stripeis which?

19. Multi-stripe Multi-frame Single-stripe Single-frame Continuum of Triangulation Methods Slow, robust Fast, fragile

20. Time-Coded Light Patterns • Assign each stripe a unique illumination codeover time [Posdamer 82] Time Space

21. Illumination history = (WB),(BW),(WB) Code Codes for Moving Scenes • Assign time codesto stripe boundaries • Perform frame-to-frametracking of correspondingboundaries • Propagate illumination history [Hall-Holt & Rusinkiewicz, ICCV 2001]

22. Designing a Code • Want many “features” to track:lots of black/white edges at each frame • Try to minimize ghosts – WW or BB “boundaries” that can’t be seen directly

23. Designing a Code 0000 1101 1010 0111 1111 0010 0101 1000 1011 0110 0001 1100 0100 1001 1110 0011 [Hall-Holt & Rusinkiewicz, ICCV 2001]

24. Implementation • Pipeline: • DLP projector illuminates scene @ 60 Hz. • Synchronized NTSC camera captures video • Pipeline returns range images @ 60 Hz. Project Code Capture Images Find Boundaries Match Boundaries Decode Compute Range

25. Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Alignment Part II: Fast ICP Done? Merging Display

26. ICP in the Real-Time Pipeline • Potential problem with ICP: local minima • In this pipeline, scans close together • Very likely to converge to correct (global) minimum • Basic ICP algorithm too slow (~ seconds) • Point-to-plane minimization • Projection-based matching • With these tweaks, running time ~ milliseconds[Rusinkiewicz & Levoy, 3DIM 2001]

27. Anchor Scans • If object moves too quickly or is out of range, ICP will fail • Global drift • Instead of aligning to last scan, keep set of scans to realign to • Large: have enough data • Distinct: don’t store too many • Allows user to restart scanning

28. Recapturing Alignment

29. Real-Time 3D Model Acquisition Pipeline 3D Scanner View Planning Alignment Part III: Voxel Grid Done? Merging Display

30. Merging and Rendering • Goal: visualize the model well enoughto be able to see holes • Cannot display all the scanned data – accumulates linearly with time • Standard high-quality merging methods:processing time ~ 1 minute per scan • Use fast voxel based merging technique

31. Merging and Rendering

32. Merging and Rendering

33. Merging and Rendering

34. + Merging and Rendering

35. Merging and Rendering • Point rendering, using accumulated normals for lighting

36. Problems with Merging • Misaligned Scans (error in ICP) • Moderate misalignment • Voxels rendered on top of one another • Noisy but OK

37. Problems with Merging • Misaligned Scans (error in ICP) • Outliers • Give wrong model • Eliminate thin and long triangles and associated points • Can be aggressive eliminating b/c of large amt data • Also rejects low quality data

38. Example: Photograph 18 cm.

39. Result

40. Postprocessing • Real-time display • Quality/speed tradeoff • Goal: let user evaluate coverage, fill holes • Offline postprocessing for high-quality models • Global registration • High-quality merging (e.g., using VRIP [Curless 96])

41. Example: Photograph 18 cm.

42. Postprocessed Model

43. Summary • 3D model acquisition pipeline optimized for obtaining complete, hole-free models • Use human’s time most efficiently • Pieces of pipeline selected for real-time use: • Structured-light scanner for moving objects • Fast ICP variant • Simple grid-based merging, point rendering

44. Limitations • Prototype noisier than commercial systems • Could be made equivalent with careful engineering • Ultimate limitations on quality: focus, texture • Scan-to-scan ICP not perfect  alignment drift • Due to noise, miscalibration, degenerate geometry • Reduced, but not eliminated, by “anchor scans” • Possibly combine ICP with separate trackers