Adaptive Skin Color Classification

1. Adaptive Skin Color Classification

2. challengeour approach results outlook Motivation Skin color detection supports… • face model fitting • mimic recognition • person identification • gaze estimation • fatigue detection (e.g. vehicle) • hand tracking • gesture recognition • action recognition • supervising work Technische Universität München Matthias Wimmer

3. challengeour approach results outlook Challenge • Skin color depends on image conditions: • illumination: light source, light color, shadow, shading,… • camera: type, settings,… • visible person: ethnic group, tan,… • Skin color occupies a large area within color space Technische Universität München Matthias Wimmer

4. challengeour approach results outlook Observations • Skin color varies greatly between images. • Skin color varies slightly within an image. Basic idea • learn image specific skin color characteristics • parameterize a skin color classifier accordingly skin color of image1 skin color of image2 Technische Universität München Matthias Wimmer

5. challengeour approach results outlook Our approach Offline step: • learn the skin color mask • specific for any face detector Online steps: • Step 1: detect the image specific skin color model • using the face detector • using the skin color mask • Step 2: adapt a skin color classifier • Step 3: calculate the skin color image Technische Universität München Matthias Wimmer

6. challengeour approach results outlook Skin color model & skin color classifier • detect the face • extract the skin color pixels • skin color model: • mean values: μr, μg, μbase • standard deviations: σr, σg, σbase • adaptive skin color classifier: skin := lowr ≤ r ≤ highrlowg ≤ g ≤ highg lowbase ≤ base ≤ highbase • learn the bounds lowr := μr – 2σrhighr := μr + 2σr . . . . . . . . . Technische Universität München Matthias Wimmer

7. challengeour approach results outlook Results • better results for • colored persons • exact shape outline • detection of facial parts:eyes, lips, brows,… • correctly detected pixels: • non-adaptive approach: 90.4% 74.8% 40.2% • adaptive approach: 97.5% 87.5% 97.0% • improvement: 7.9% 17.0% 141.3% Technische Universität München Matthias Wimmer

8. challengeour approach results outlook Conclusion • Challenge: much variation within skin color • illumination, camera, visible person • skin color occupies a large area within color space • We propose a way to reduce those variations • exploit an image specific skin color model • adapt a skin color classifier to that skin color model • We proved our approach • using a simple but real-time capable skin color classifier • comparison: non-adaptive ↔ adaptive Technische Universität München Matthias Wimmer

9. challengeour approach results outlook Ongoing research • Learn skin color mask for other face detectors • Specialize more powerful skin color classifiers • Recognize other feature images/color images • lip color image • tooth color image • eye color image • hair color image • eye brow color image example: lip color detection Technische Universität München Matthias Wimmer

10. Thank you ! Technische Universität München Matthias Wimmer

11. Overview • Motivation / Challenge • Our approach • extracting skin color pixels • adapt skin color classifier • Results • Conclusion • Outlook Technische Universität München Matthias Wimmer

12. challengeour approach results outlook Challenge (2): non-skin color pixels • Skin color pixels have to be separated from non-skin color pixels. • Areas of skin color and non-skin color overlap. • Color can not make a distinctive separation. Technische Universität München Matthias Wimmer

13. Basic idea • learn image specific skin color characteristics • parameterize classifier accordingly with those characteristics Technische Universität München Matthias Wimmer

14. challengeour approach results outlook Offline: Learn the skin color mask 1. 2. 3. • face image database with labeled skin color pixels • skin color mask: array with 24 x 24 cells Computational steps: • detect the face in every image • every cell is assigned the relative number of labeled skin color pixels at its position • apply threshold Technische Universität München Matthias Wimmer

15. challengeour approach results outlook Step 1: Detect the image specific skin color model • detect the face • extract the skin color pixels • normalized RGB color space: base = R + G + B r = R / base g = G / base • skin color model: • mean values: μr, μg, μbase • standard deviations: σr, σg, σbase Technische Universität München Matthias Wimmer

16. challengeour approach results outlook Step 2: Adapt a skin color classifier • non-adaptive skin color classifier: skin := 0.35 ≤ r ≤ 0.5 0.2 ≤ g ≤ 0.7 200 ≤ base ≤ 740 • adaptive skin color classifier: skin := lowr ≤ r ≤ highrlowg ≤ g ≤ highglowbase ≤ base ≤ highbase • learn the bounds via the skin color model • mean value and standard deviationlowr := μr – 2σrhighr := μr + 2σr . . . . . . . . . • linear function:lowr := aμr + bμg + cμbase + dσr + eσg + fσbase + g . . . Technische Universität München Matthias Wimmer

17. Adaptive skin color classifier • non adaptive skin color classifier: skin := 0.35 ≤ r ≤ 0.5  0.2 ≤ g ≤ 0.7  200 ≤ base ≤ 740 • adaptive skin color classifier: skin := lowr ≤ r ≤ highr lowg ≤ g ≤ highg lowbase ≤ base ≤ highbase • learn the bounds out of the skin color model Technische Universität München Matthias Wimmer

18. challengeour approach results outlook Related work • Feedback of information fromhigh level vision components to low level vision components Technische Universität München Matthias Wimmer