N=billions: The smartphone revolution in the behavioral sciences

1. N=billions: The smartphone revolution in the behavioral sciences Geoffrey Miller

2. Almost everyone in the world will soon be able to participate in any behavioral research – remotely, electronically, continually

3. 2012: • 7.0 billion people total • >5.0 billion mobile phone users • 1.1 billion in China • 900 million in India • 700 million in Africa Smartphone users: • 2012: 1.1 billion • 2015: 3 billion • 2020: 5 billion (ITU, 2012; Erickson, 2012)

5. Smartphone R&D: A windfall for behavioral sciences Revenue 2012: • $1.5 trillion telecomms services • $200 billion smartphone sales • cf. Pharmaceuticals $970 billion Smartphone R&D spending 2012 • Samsung $6 billion • Nokia $1.7 billion • Apple $1 billion cf. behavioral science research: • NIMH $1.5 billion • NSF $250 million

6. Ideal method for gathering data if psychology was invented today? Smartphones. general-purpose computer + sensor array + GPS system + media capture system (+ small enough to hold to your head for phone calls) Ubiquitous, unobtrusive, intimate Remotely accessible through participants downloading ‘psych apps’ Can gather precise, objective, sustained, ecologically valid data on real-world behaviors & experiences of millions

7. Smartphone research methods • “The smartphone psychology manifesto” (2011), Perspectives on Psychological Science • Symposia at: • Society for Experimental Social Psychology (Austin Oct 2012) • Association for Psychological Science (D.C., May 2013) • Association for Consumer Research (Chicago Oct 2013)

8. What can smartphones do for psychology? Example: Samsung Galaxy S3 Released June 2012 Sold 20m within 3 months $1 bn/month in profits

9. Samsung Galaxy S3 Carried throughout day Small: 5.4 x 2.8 x 0.3” Light: 4.7 ounces Reliable: 43 hour battery life Continual data-gathering: sensors, GPS, app logs, call logs Familiar, intimate, personalized, trusted, unobtrusive habituation  low reactance ecological validity

10. Samsung Galaxy S3Android 4.1 operating system Apps can run in background w/o annoying subjects Develop & download “psych apps” for studies msec timing unlike Mturk(e.g. Dufau2011 lexical decision app N=4,157) Software access to all other apps: phone calls, emails, web browser, Facebook “Root access” to all hardware, sensors

11. Samsung Galaxy S3

12. Samsung Galaxy S3pocket supercomputer ARM Cortex A9 Chip with Exynos Quad-core 1.4 GHz CPU Faster than your office PC 21 Mbps 4G Broadband  access cloud computing for unlimited processing & memory Can run sophisticated AI, speech recognition, psych apps

13. Samsung Galaxy S3personal data repository 2 GB RAM, 64 GB flash memory Medical records Genomes MRIs School records Legal records Spending patterns Credit history Contact lists Photo albums Music tastes Home videos

14. Samsung Galaxy S3Input/Output for running surveys, experiments Dual mikes, noise cancelling, voice recognition 1280x720 pixel 4.8” touchscreen Button taps, gestures, virtual keyboard Rear 8 MP camera (1080p@30fps video) Front 1.9 MP camera (720p@30fps video) Smart Stay eye tracking

15. Samsung Galaxy S3Input/Output for high-quality audio & video 3.5mm stereo jack microUSB2.0, MHL to HDMI to 3-D HDTV video & 8-channel digital audio

16. Samsung Galaxy S3Connectivity: download psych apps, upload data, sense environment & interactions Bluetooth, WiFi: sense nearby electronics 4G broadband internet (HSPA+ @ 21 Mbit/s) NFC payments, purchases

17. Samsung Galaxy S3Onboard sensors: location, 300 GPS + 24 GLONASS navsats: +/- 10m lat/long,10 nsec time • GISdigital maps • GPS receiver chip • life-tracking, spatial behavior, context-aware experiments

18. Samsung Galaxy S3Onboard sensors: movements 3-axis accelerometer: motion tracking, activity type, energy output, health, mood 3-axis gyroscope: orientation tracking Barometer: altitude, weather Digital compass: direction tracking

19. External sensors Wireless intraocular glaucoma pressure sensor (1 cubic millimeter) Consumer EEG headsets, e.g,. Emotiv EPOC 14-channel ($300) Future biological microelectromechanical systems (BioMEMS): wearable, implanted, injected to monitor temp, blood pressure, pulse, blood alcohol & drugs, hormones, immune system, inflammation, stress, ovulation, brain function Remote neuropsych, health psych, substance abuse research

20. External biosensors:Continua Health Alliance • Trade association of 220 companies • Promotes mHealth, certifies Bluetooth & Zigbee biosensors – heart ECG, blood glucose monitors, ultrasound imagers • Promoters: Samsung, Sharp, Panasonic, Sony, Fujitsu, Intel, IBM, Texas Instruments, Oracle, Cisco, Orange, Qualcomm, Novartis, Roche, UnitedHealth

21. mHealth = health care & research through mobile phones • open-source mHealthspecs, standards, software for: • Surveys, biosensors, RFID med tracking, behavior tracking (GPS, sensors, call logs) • Continuous symptom monitoring integrated across diseases • Whole-population drug trials, N-of-1 longitudinal trials • Current projects: PTSD, pain management • Want to work with psychologists: behavioral data sensing & standards Deborah Estrin Ida Sim

22. Practicalities Technical challenges Conflicting apps Limited battery power (esp. for GPS) Limited-accuracy sensors (but getting better) Heat dissipation (if psych app is CPU -hungry) Unreliable telecoms service in some countries Participant behavior challenges Forgetting to charge & carry smartphone daily Losing or upgrading phones during study Lending phones to others Malicious hackers Reactivity & self-consciousness

23. Practicalities Global recruitment potential, but constrained by • Selection biases: who owns smartphones • Smartphone OS, drivers, hardware specs • Language of app interface • Geoareas covered by GIS databases • Payment systems, exchange rates Programming psych apps • Software development kids (devkits) –assume knowing Java, Objective-C, C#, etc. • No easy-to-use psych app devkits yet • Collaborate with computer scientists, user experience researchers, smartphone manufacturers, telecoms service providers

24. Data analysis challenges NikoKiukkonen et al. (2010) at Nokia: Tracked 168 smartphone users 4 months each Data flood: 15 million Bluetooth scans of nearby devices 13 million Wi-Fi scans of nearby routers 5 million GPS records 4 million app usage records 500,000 accelerometer readings 220,000 audio samples 130,000 voice calls 90,000 text messages 28,000 photos taken 2,000 videos shot

25. Psych will have develop new ‘behavioral informatics’ methods = 1080p 30fps Pivothead recording glasses, $350 CERN LHC computing grid: 300 MB/s raw data output, handled by 200,000 processors and 150,000 TB disk space across 34 countries; 15 petabytes/year = data output from 70 participants recording one hour of HD video per day

26. Human subjects & IRB challenges Anonymity impossible given GPS, app usage, call logs, sensor logs Truly informed consent hard to get Software licensing agreements rarely read – Google, Facebook, Apple data-mining Confidentiality vulnerable to authorities, telecoms service providers & hardware manufacturers (e.g. Huawei) Rules unclear given global recruitment, access to vulnerable recruits, data storage with cloud computing Potential for liability, fraud, identity theft, blackmail

27. Smartphones vs. other research methods Pros: Potential global recruitment & very large samples High convenience, ecological validity, unobtrusiveness Easy video/audio capture, motion sensing, location tracking Potential HD video/audio given common peripherals Potential remote neurophysiology & health psych given Bluetooth biosensors

28. Smartphones vs. other research methods Cons: Substantial study prep work writing & field-testing the psych app Low contextual control over participant’s social & physical environment Potentially very large datasets requiring new behavioral informatics methods Ethical challenges in informed consent, anonymity, confidentiality, liability, & rule ambiguity given global recruitment & cloud data storage

29. Challenges for Berkman? • Legal & moral issues: human subjects protection, privacy, data security • Economic issues: payment systems • Social issues of behavioral Big Data & personal data

30. Conclusions A tech windfall for psychology Telecoms R&D $billions creating a pervasive, unified, global, context-aware system for sensing behavior & experience of almost everyone on earth Smartphones will quickly grow even more powerful & ubiquitous We just have to learn how to tap into the data-torrent Questions?

31. Extra slides for answering questions

32. Big Data in psychology YourMorals.org N=thousands N > 100,000 (moral values: Jesse Graham et al., 2010) N=61m (political influence: Robert Bond et al., 2012 , Nature)

33. Previous research using mobile electronic devices • Analyze anonymized telecoms call-routing records: movements & social connections of millions of people • Distribute preprogammed PDAs, EARs: experience sampling, diary studies, conversation samples • Distribute preprogammed smartphones: MIT Reality Mining project • Distribute apps remotely to existing smartphones: MyExperience, Mappiness • Recruit any user, anywhere • Huge samples • Rich data

34. Smartphones: all-purposeelectronic Swiss army knifes Replacing: landline phones, digital cameras, photo books, video recorders, MP3 music players, radios, voice recorders, GPS navigators, handheld game consoles, watches, alarm clocks, calendars, and calculators Can replace paper-and-pencil surveys, mail surveys, phone surveys Given the right peripherals, could replace many lab studies, field studies, and Internet studies

35. What will smartphones do soon? RFID chip readers, can sense all tagged objects nearby (30 bn RFID tags produced per year; soon > 1 tr): consumer psych Better picoprojectors (e.g. Samsung Beam): group experiments Electronic glasses & contact lenses, augmented reality apps & experiments Better speech recognition Bluetooth haptic, kinematic, mo-cap peripherals (like Xbox 360 Kinect) Piezoelectric clothing to charge batteries, record joint movements

36. What will smartphones do soon? By 2016: 8-core 2-GHz CPUs, 64-core GPU for 3D HD video Moore’s law: transistor density on chips (computer speed) doubles every two years

37. Smartphones as handheld supercomputers = 1997: Intel ASCI Red, Sandia Nat’l Labs: first teraflop supercomputer 9,298 333-MHz processors, 104 cabinets, 2,500 square feet, $50 million Within 10 years: 64-core 50-GHz processors … plus unlimited cloud computing power, c. $500

38. Kryder’s law: memory storage capacity doubles every 12 months 16 GB micro SD flash memory cards 2012: 32 GB flash memory chip in Galaxy Nexus 2025: >200 TB memory in smartphones

39. Connectivity: Nielsen’s law: Internet connection speeds increase 50% per year 2012: HTC Thunderbolt = 300 Mbit/s LTE broadband 2025: 60 Gbit/s = 5x HDMI 1.3 cable = drive 5 HD digital cinema projectors

40. From phone-centric to data-centric devices 1987: ‘Wall Street’ phone 1993: first smartphone IBM ‘Simon’ 2007: Apple iPhone 2008: Android OS

41. Evolution of the mobile phone1983 - 2012

42. Heat dissipation: Koomey’s law: computations per kWh energy dissipated increases 50% per year Keep smartphones from getting too warm

45. Abstract • 5.9 billion people now use mobile phones, of which 1.1 billion are smartphones.  Smartphones will empower behavioral scientists to collect terabytes of ecologically valid data from vast global samples – easily, quickly, and remotely.  Smartphones can record where people are, what they are doing, and what they can see and hear. They can run interactive surveys, tests, and experiments through touch screens and Bluetooth peripherals.  This talk focuses on what smartphones can do now, and will be able to do in the near future, as research platforms. Smartphone research will require new skills in app development, Big Data analysis, and recruitment through social media, and will raise tough new ethical issues, but smartphones could transform the behavioral sciences even more profoundly than PCs and brain imaging did. By 2025, billions of potential research participants will be carrying ultra-broadband, sensor-rich smartphones with GPS, augmented reality goggles, and biosensors that allow remote psychophysiology. These will render some current research methods obsolete, and will open extraordinary new opportunities for understanding human nature and culture.