Sharpening from Shadows: Sensor Transforms for Removing Shadows using a Single Image

1. Sharpening from Shadows: Sensor Transforms for Removing Shadows using a Single Image School of Computer Science Simon Fraser University November 2009

2. Outline • Image Formation • Invariant Image Formation • Finding invariant direction by calibration • Finding invariant direction by minimizing entropy • Sharpening Matrix • Proposed Method • Optimization problem • Result

3. Shadow Removal Method • To generate shadowless images, there are two steps: • Finding Illuminant Invariant image (grayscale) • Creating colored shadowless images using edges in main image and invariant image. [Finlayson et al. (ECCV2002)] Main Image Invariant Image Shadowless Image This Paper

4. Image Formation Light spectral Camera sensitivity Surface reflection Camera Response:

5. Image Formation Simplification • Camera sensors represented as delta functions. • Illumination is restricted to the Planckian locus. • Wien’s approximation for temperature range 2500K to 10000K. We have:

6. Invariant Image Formation • Using the simplified model we form band-ratio chromaticities rk by dividing Rand B by Gand taking the logarithm: • As temperature T changes, 2d-vectors rk ,k=R,B, will follow a straight line in 2d chromaticity space. For all surfaces, the lines will be parallel, with slope (ek –eG). Camera Dependent Surface Dependent

7. Invariant Image Formation • The invariant image, then, is formed by projecting 2-d colors into the direction orthogonal to the 2-vector (ek –eG). • So, the problem is reduced to finding the direction. • Why we are interested? • Shadow is nothing just the surface in different illumination condition. (they should be in a line)

8. Finding Invariant Direction • Calibrating Camerato find the invariant direction. [Finlayson et al. (ECCV2002)] • Need many images under different illumination. • Good for camera company not images with unknown camera. HP912 Digital Still Camera: Log-chromaticities of 24 patches; 7 patches, imaged under 9 illuminants.

9. Finding Invariant Direction • Without calibrating the camera, can use entropy of projection to find the invariant direction [Finlayson et al. (2004)]: Wrong direction – higher entropy Correct direction – smaller entropy

10. Sharpening Transform Matrix • Convert a given set of sensor sensitivity functions into a new set that will improve the performance of any color-constancy algorithm that is based on an independent adjustment of the sensor response channels. • Transform the camera sensors to made them more narrow band, which is one of the assumption that we made. • It also could apply to the image instead of sensors.

11. Proposed Method 1 • Select shadow and non shadow pixels for the same surface material. • Find the sharpening matrix which makes the chromaticities of selected pixels as linear as possible in log-log plane = an optimization problem. • Transform the main image by sharpening matrix. • Create illumination invariant image by entropy-minimization method [Finlayson et al. (2004)]. 2 .7 0.15 .15 .15 .70 .15 .15 .15 .70 3 4

12. Shadow and Non Shadow Regions • The user selects the shadow and non shadow region of a surface. • For future work this could be automatic . • According to invariant formation in ideal condition, the chromaticities of these point in log-log plane should be in a line. User Defined

13. Optimization Problem • To find best sharpening matrix M3x3 in order to make the chromaticity as linear as possible: m11 m12 m13 m21 m22 m23 m31 m32 m33 sum is 1 Linear combination more than 1-β Colors don’t change completely

14. Objective Function • F return the minimum entropy of log chromaticities projected to all directions. • rank is meant to encourage a non-rank-reducing matrix M. Log chromaticities entropy Minimum entropy For this M

15. Sharpening Matrix Shadow and non shadow region chromaticity Less linear Sharpening Matrix More linear

16. Results Original Sharpened Invariant Difference

17. Good vs. poor sharpening matrix .75 -.20 .02 .01 .86 .13 .24 .34 .84 Obj. Func. = .0942 More linear .90 .30 -.14 -.04 .79 .16 .14 -.09 .98 Obj. Func. = .0487 minimum

18. Result

19. Result

20. Conclusion • We proposed a new schema for generating illumination invariant for removing shadow. • The contribution of this paper is using sharpener matrix to get better shadow removal. • The method use single images which is more practical compared to camera calibration methods which needs bunch of images in different illumination condition.

21. References • Sharpening Matrix:G.D. Finlayson, M.S. Drew, and B.V. Funt. Spectral sharpening: sensor transformations for improved color constancy. J. Opt. Soc. Am. A, 11(5):1553–1563, May 1994. • Illumination invariant image:G.D. Finlayson, S.D. Hordley, and M.H. Brill. Illuminant invariance at a single pixel. In 8th Color Imaging Conference: Color, Science, Systems and Applications., pages 85–90, 2000. • Shadow removal method:G.D. Finlayson, S.D. Hordley, and M.S. Drew. Removing shadows from images. In ECCV 2002: European Conference on Computer Vision, pages 4:823–836, 2002. Lecture Notes in Computer Science Vol. 2353. • Entropy minimization method:G.D. Finlayson, M.S. Drew, and C. Lu. Intrinsic images by entropy minimization. In ECCV 2004: European Conference on Computer Vision, pages 582–595, 2004. Lecture Notes in Computer Science Vol. 3023.

22. Questions? Thank you. Thanks! To Natural Sciences and Engineering Research Council of Canada