SurroundSense: Mobile Phone Localization via Ambience Fingerprinting

1. SurroundSense: Mobile Phone Localization via Ambience Fingerprinting

2. Context Pervasive wireless connectivity + Localization technology = Location-based applications

3. Location-Based Applications (LBAs) For Example: GeoLife shows grocery list when near Walmart MicroBlog queries users at a museum Location-based ad: Phone gets coupon at Starbucks iPhone AppStore: 3000 LBAs, Android: 500 LBAs

4. Most emerging location based apps do not care about the physical location GPS: Latitude, Longitude

5. Most emerging location based apps do not care about the physical location Instead, they need the user’s logical location GPS: Latitude, Longitude Starbucks, RadioShack, Museum, Library

6. Physical Vs Logical Unfortunately, most existing solutions are physical GPS GSM based SkyHook Google Latitude RADAR Cricket …

7. Given this rich literature, Why not convert from Physical to Logical Locations?

8. Physical Location Error

9. Starbucks Pizza Hut Physical Location Error

10. Starbucks Pizza Hut Physical Location Error The dividing-wall problem

11. SurroundSense: A Logical Localization Solution

12. Hypothesis It is possible to localize phones by sensing the ambience such as sound, light, color, movement, WiFi …

13. Hypothesis It is possible to localize phones by sensing the ambience such as sound, light, color, movement, WiFi …

14. Multi-dimensional sensing extracts more ambient information Any one dimension may not be unique, but put together, they may provide a unique fingerprint

15. SurroundSense • Multi-dimensional fingerprint • Based on ambient sound/light/color/movement/WiFi Starbucks Pizza Hut Wall

16. Should Ambiences be Unique Worldwide? J P I H A Q B K C D L Q E M N R F O G

17. Should Ambiences be Unique Worldwide? GSM provides macro location (strip mall) SurroundSense refines to Starbucks J P I H A Q B K C D L Q E M N R F O G

18. Why does it work? The Intuition: Economics forces nearby businesses to be diverse Not profitable to have 3 adjascent coffee shops with same lighting, music, color, layout, etc. SurroundSense exploits this ambience diversity

19. SurroundSense Architecture Matching Ambience Fingerprinting Sound Test Fingerprint Color/Light + Acc. = WiFi Logical Location Fingerprint Database GSM Macro Location Candidate Fingerprints

20. Fingerprints Acoustic fingerprint (amplitude distribution) 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 • Sound: (via phone microphone) • Color: (via phone camera) Normalized Count -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 Amplitude Values Color and light fingerprints on HSL space 1 0.5 0 Lightness 0 1 0.8 0.5 0.6 0.4 0.2 Hue 1 0 Saturation

21. Fingerprints • Movement: (via phone accelerometer) Grocery Store Cafeteria Clothes Store Moving Static

22. Fingerprints • Movement: (via phone accelerometer) Grocery Store Cafeteria Clothes Store Moving Static Queuing Seated

23. Fingerprints • Movement: (via phone accelerometer) Grocery Store Cafeteria Clothes Store Moving Static Short walks between product browsing Pause for product browsing

24. Fingerprints • Movement: (via phone accelerometer) Grocery Store Cafeteria Clothes Store Moving Static Walk more Quicker stops

25. Fingerprints • Movement: (via phone accelerometer) • WiFi: (via phone wireless card) Grocery Store Cafeteria Clothes Store Moving Static ƒ(overheard WiFi APs)

26. Discussion • Time varying ambience • Collect ambience fingerprints over different time windows • What if phones are in pockets? • Use sound/WiFi/movement • Opportunistically take pictures • Fingerprint Database • War-sensing

27. Evaluation Methodology • 51 business locations • 46 in Durham, NC • 5 in India • Data collected by 4 people • 12 tests per location • Mimicked customer behavior

28. Evaluation: Per-Cluster Accuracy Localization accuracy per cluster Accuracy (%) Cluster

29. Evaluation: Per-Cluster Accuracy Localization accuracy per cluster Fault tolerance Accuracy (%) Cluster

30. Evaluation: Per-Cluster Accuracy Sparse WiFi APs Localization accuracy per cluster Accuracy (%) Cluster

31. Evaluation: Per-Cluster Accuracy Localization accuracy per cluster Accuracy (%) No WiFi APs Cluster

32. Evaluation: Per-Scheme Accuracy

33. Evaluation: User Experience Random Person Accuracy 1 WiFI Snd-Acc-WiFi Snd-Acc-Clr-Lt SurroundSense 0.9 0.8 0.7 CDF 0.6 0.5 0.4 0.3 0.2 0.1 0 0 10 20 30 40 50 60 70 80 90 100 Average Accuracy (%)

34. Limitations and Future Work Energy-Efficiency • Continuous sensing likely to have a large energy draw

35. Limitations and Future Work Energy-Efficiency • Continuous sensing likely to have a large energy draw Localization in Real Time • User’s movement requires time to converge

36. Limitations and Future Work Energy-Efficiency • Continuous sensing likely to have a large energy draw Localization in Real Time • User’s movement requires time to converge Non-business locations • Ambiences may be less diverse

37. Limitations and Future Work • Electroic compasses can fingerprint layout • Tables and shelves laid out in different orientations • Users forced to orient in those ways

38. Ambience can be a great clue about location Ambient Sound, light, color, movement … None of the individual sensors good enough Combined they may be unique Uniqueness facilitated by economic incentive Businesses benefit if they are mutually diverse in ambience Ambience diversity helps SurroundSense Current accuracy of 89% Conclusion

39. Questions?

40. SurroundSense evaluated in 51 business locations Achieved 89% accuracy But perhaps more importantly, SurroundSense scales to any part of the world Summary

41. SurroundSense • Today’s technologies cannot provide logical localization • Ambience contains information for logical localization • Mobile Phones can harness the ambience via sensors • More sensors, beter reliability • Evaluation results: • 51 business locations, • 87% accuracy SurroundSense can scale to any part of the world

43. Evaluation: Per-Shop Accuracy Localization Accuracy per Shop 1 WiFi Snd-Acc-WiFi Snd-Acc-Clr-Lt SurroundSense 0.9 0.8 0.7 0.6 0.5 CDF 0.4 0.3 0.2 0.1 0 0 10 20 30 40 50 60 70 80 90 100 Average Accuracy (%)

44. Architecture: Filtering & Matching Candidate Fingerprints

45. Fingerprinting Sound • Fingerprint generation : Signal amplitude • Amplitude values divided in 100 equal intervals • Sound Fingerprint = 100 normalized values • valueX = # of samples in interval x / total # of samples • Filter Metric: Euclidean distance • Discard candidate fingerprint if metric > threshold г • Threshold г • Multiple 1 minute recordings at the same location • di = max dist ( any two recordings ) • г = max ( di of candidate locations )

46. Fingerprinting Motion • Fingerprint generation: stationary/moving periods • Sitting (restaurants, cafes, haircutters ) • Slow Browsing (bookstores, music stores, clothing) • Speed-Walking (groceries) • Filter Metric: • Discard candidate fingerprints with different classification • 0.0 ≤ R ≤ 0.2 sitting • 0.2 ≤ R ≤ 2.0 slow browsing • 2.0 ≤ R ≤ ∞ speed-walking

47. Fingerprinting WiFi • Fingerprint generation: fraction of time each unique address was overheard • Filter/Ranking Metric • Discard candidate fingerprints which do not have similar MAC frequencies

48. Fingerprinting Color • Floor Pictures • Rich diversity across different locations • Uniformity at the same location • Fingerprint generation: pictures in HSL space • K-means clustering algorithm • Cluster’s centers + sizes • Ranking metric

49. Thinking about Localization from an application perspective…

50. Emerging location based apps need place of user, not physical location Starbucks, RadioShack, Museum, Library Latitude, Longitude