Recognizing Surfaces using Three-Dimensional Textons

1. Recognizing Surfaces using Three-Dimensional Textons Thomas Leung and Jitendra Malik Computer Science Division University of California at Berkeley

2. Traditional Texture Recognition • Assume texture to be planar; • Assume constant illumination and viewing directions; • Ignore 3D nature of natural materials, i.e. no shadowing, occlusions, etc… • E.g. Puzicha et al, Jain et al, Greenspan et al, etc…. ICCV '99, Corfu, Greece

3. Example Natural Materials Terrycloth Rough Plastic Plaster-b Sponge Rug-a Painted Spheres Columbia-Utrecht Database (http://www.cs.columbia.edu/CAVE) ICCV '99, Corfu, Greece

4. Materials under different illumination and viewing directions Different illumination and viewing directions Plaster-a Crumpled Paper Concrete Plaster-b (zoomed) ICCV '99, Corfu, Greece

5. Task Felt? Polyester? Terrycloth? Rough Plaster? Leather? Plaster? Concrete? Crumpled Paper? Sponge? Limestone? Brick? ? ? ICCV '99, Corfu, Greece

6. 3D Texture Models • Analytical models: • Simple parametric surface height distribution; • compute image statistics; • Dana & Nayar 97, 98, 99; Koenderink et al 96, 98; Leung & Malik 97; Chantler et al 97, 98; • Computer graphics models: • bump maps, displacement maps, point clouds, etc. • difficult to obtain for natural materials; ICCV '99, Corfu, Greece

7. Problem Formulation Image Database Task Recognize new sample of different light/view ICCV '99, Corfu, Greece

8. Main Idea • Natural materials are made up of local features (geometric and photometric); • There exists a universal set of local features for all materials; • How these local features change appearance with different illumination and viewing directions determine how the materials look. ICCV '99, Corfu, Greece

9. Outline • Learning the universal vocabulary of local structures • Material models • Results ICCV '99, Corfu, Greece

10. Outline • Learning the universal vocabulary of local structures • Introduce 2D textons for planar texture; • Extend to 3D textons for natural materials; • Material models • Results ICCV '99, Corfu, Greece

11. 2D Textons • Julesz suggests a universal vocabulary for such features --- textons [Julesz 81]; • crossings, line-ends, junctions, etc… • Define textons for real images. ICCV '99, Corfu, Greece

12. 2D Textons • Goal: find canonical local features in a texture; 1) Filter image with linear filters: 2) Vector quantization on filter outputs; 3) Quantization centers are the textons. • Spatial distribution of textons defines the texture; ICCV '99, Corfu, Greece

13. 2D Textons (cont’d) ICCV '99, Corfu, Greece

14. 3D Textons • Consider textures with 3D features, e.g. bumps, grooves, ridges, etc… • Want textons to capture local 3D geometric and photometric features; • One image is ambiguous: different features can look the same under certain illumination and viewing conditions; • More images will discriminate between the different cases. ICCV '99, Corfu, Greece

15. Learning 3D Textons Rough Plastic Concrete 3D textons Light/view 1 Texton 1 Light/view 2 Texton 2 Texton K Light/view N ICCV '99, Corfu, Greece

16. Algorithm for Learning Vocabulary • Register all 20 images for each material; • Filter images with filter bank of 48 kernels; • Concatenate filter responses of the 20 images; • Each pixel becomes a 960 (20x48) dimensional feature vector; • Apply K-means to the feature vectors of all materials together; • Resulting centers are the 3D textons. ICCV '99, Corfu, Greece

17. Algorithm for 3D Textons ICCV '99, Corfu, Greece

18. Universal 3D Texton Vocabulary • Columbia-Utrecht Database (60 materials, each with 205 images) • Vocabulary of textons learned from 20 training materials; • Use 20 different light/view images for each material. ICCV '99, Corfu, Greece

19. Examples of 3D Textons Different illumination and viewing directions Texton 1 Texton 2 Texton 3 Texton 4 Texton 5 Texton 6 Texton 7 ICCV '99, Corfu, Greece

20. Quantization Errors • Reconstruct images after quantization; • SSD error within 5%. ICCV '99, Corfu, Greece

21. Outline • Learning the universal vocabulary of local structures; • Material models; • Image to texton representation; • Material representation using textons; • Results. ICCV '99, Corfu, Greece

22. Texton Labeling • Each pixel labeled to texton i (1 to K) which is most similar in appearance; • Similarity measured by the Euclidean distance between the filter responses; ICCV '99, Corfu, Greece

23. Material Representation • Each material is now represented as a spatial arrangement of symbols from the texton vocabulary; • Recognition --- ignore spatial arrangement, use histogram (K=100); ICCV '99, Corfu, Greece

24. Histogram Models for Recognition Rough Plastic Pebbles Plaster-b Terrycloth ICCV '99, Corfu, Greece

25. Similarity of materials • Similarity between histograms measured using chi-square difference: ICCV '99, Corfu, Greece

26. Similarity Matrix Plaster-a Plaster-b Aluminum Foil Cork ICCV '99, Corfu, Greece

27. Outline • Learning the universal vocabulary of local structures • Material models • Results • Material recognition from single image; • Synthesis of novel images. ICCV '99, Corfu, Greece

28. Recognition from Single Image • 4 images to build histogram for model; • 1 image of novel illumination and/or viewing directions to be recognized; Image Database Novel image ? ICCV '99, Corfu, Greece

29. Novel Image from Material i? • Build texton histogram for novel image. • Compare with texton histogram for material i. • However, texton labeling from 1 image is difficult, because in 1 light/view, several textons may have same appearance. • Each pixel has N possible texton labels; • Need to find the labeling that maximizes Similarity(novel image, material i) ICCV '99, Corfu, Greece

30. Markov chain Monte Carlo for finding labeling • Randomly label each pixel to one of N possibilities. Call this the initial state x(t),t=0 • Compute P(x(t)|material i); • Obtain x’ by randomly changing M labels of x(t); • Compute P(x’|material i); • Compute • If , the x’ is accepted, otherwise, accept with probability . ICCV '99, Corfu, Greece

31. P(detection) vs P(false alarm) ICCV '99, Corfu, Greece

32. Synthesis of images with novel illumination and viewing directions Map each pixel to textons Textons tell us how appearance changes ICCV '99, Corfu, Greece

33. Synthesis of novel light/view images • Keep exact spatial arrangement of textons ICCV '99, Corfu, Greece

34. Synthesis Results Plaster-a Concrete Texture Mapping Texture Mapping Ground Truth 3D Texton Model Ground Truth 3D Texton Model ICCV '99, Corfu, Greece

35. Synthesis Results Crumpled paper Plaster-b (zoomed) Texture Mapping Texture Mapping Ground Truth 3D Texton Model Ground Truth 3D Texton Model ICCV '99, Corfu, Greece

36. Synthesis Results Rough Plastic Sponge Texture Mapping Texture Mapping Ground Truth 3D Texton Model Ground Truth 3D Texton Model ICCV '99, Corfu, Greece

37. Similarity to Appearance-based Object Recognition • Object Recognition: objects are represented by a collection of images under different illumination and viewing conditions; • Material Recognition: materials are represented by 3D textons, each of which is represented by the appearances under different illumination and viewing conditions. ICCV '99, Corfu, Greece

38. Conclusions • Model natural materials through images; • Learn a universal vocabulary of 3D textons; • Use the vocabulary to • recognize materials from a single image of novel illumination and viewing directions; • synthesize materials at novel illumination and viewing directions. ICCV '99, Corfu, Greece