Appearance Models

1. Appearance Models • Shape models represent shape variation • Eigen-models can represent texture variation • Combined appearance models represent both

2. Appearance Models • Statistical model of shape and texture • Generative model • general • specific • compact

3. Building Appearance Models • For each example extract shape vector • Build statistical shape model, Shape, x = (x1,y1, … , xn, yn)T

4. Building Appearance Models • For each example, extract texture vector Shape, x = (x1,y1, … , xn, yn)T Texture, g Warp to mean shape

5. Warping texture • Problem: • Given corresponding points in two images, how do we warp one into the other? • Two common solutions • Piece-wise linear using triangle mesh • Thin-plate spline interpolation

6. Interpolation using Triangles Region of interest enclosed by triangles. Moving nodes changes each triangle Just need to map regions between two triangles

7. Barycentric Co-ordinates

8. Barycentric Co-ordinates Three linear equations in 3 unknowns

9. Interpolation using Triangles • To find out where each pixel in new image comes from in old image • Determine which triangle it is in • Compute its barycentric co-ordinates • Find equivalent point in equivalent triangle in original image • Only well defined in region of `convex hull’ of control points

10. Thin-Plate Spline Interpolation • Define a smooth mapping function (x’,y’)=f(x,y) such that • It maps each point (x,y) onto (x’,y’) and does something smooth in between. • Defined everywhere, even outside convex hull of control points

11. Thin-Plate Spline Interpolation • Function has form

12. Building Texture Models • For each example, extract texture vector • Normalise vectors (as for eigenfaces) • Build eigen-model Warp to mean shape Texture, g

13. Face Texture Model

14. Textured Shape Modes Generate position of control points Warp mean texture image (Mean points go to new points, X) Shape variation (texture fixed)

15. Textured Shape Model

16. Combined Models • Shape and texture often correllated • When smile, shadows change (texture) and shape changes • Learning this correlation leads to more compact (and specific) model

17. Learning Correlations Model accounting for correlations between shape and texture Model assuming shape and texture independent

18. Learning Correlations • For each image in training set we have best fitting shape and texture param.s • Construct new vector, • Apply PCA (mean + eigenvec.s of covar.)

19. Combined Appearance Models Varying c changes both shape and texture

20. Combined Appearance Model • Generate shape, X, and texture, g • Warp texture so mean control points lie on new X

21. Face Appearance Model

22. Face Appearance Model

23. Sub-cortical structures • 72 examples • 123 points • 5000 pixel model Caudate Nucleus Lentiform Nucleus Ventricles

24. Shape and Texture Modes Shape variation (texture fixed) Texture variation (shape fixed)

25. Combined Appearance Model • Shape and texture correlated

26. Full brain slice Shape: Texture:

27. Full brain slice Combined Mode 1 Combined Mode 2

28. Problems with viewpoint • Models require all points visible • Sometimes a problem for 2D images of 3D objects • Small rotations (+/-30o) of face modelled well • Large rotations cause occlusions • Eg eye hidden behind nose etc • Solutions • Use multiple `view based’ 2D models • Use a full 3D model

29. View-Based Models • Build 3 distinct models • Exploit symmetry Profile Profile (Reflected) Half-Profile Half-Profile (Reflected) Frontal

30. Face Profile Model Mode 1: Mode 2:

31. Half-Profile Model Mode 1: Mode 2:

32. 3D Models • Use 3D shape model (3n-D vectors) • Points control a polyhedral mesh • Texture mapped onto mesh and modelled • Reconstruct by generating new texture and mapping onto 3D mesh described by shape model

33. 3D Models Mesh = + Texture

34. Interpreting Images (1) Place model in image Measure Difference Update Model Iterate